如何修复“.htaccess RewriteRule 导致页面重复”
How do I fix ".htaccess RewriteRule results in duplicated pages"
美好的一天。
我们有一个用 vanilla PHP 编码的门户网站，其中有一个博客部分，其中
[mysite.com/blog.php?blog=1] 输出所需文件的内容。
这导致我们的 SEO 专家指出它的格式不适合 SEO URL。
然后我们决定使用 .htaccess 来显示名为 URLs
blog=Residential_Relocation -> blogs.php?blog=1输出[mysite.com/blog.php?blog=Residential_Relocation]
但现在它被视为重复。
我们如何才能只从 blog=1 URL 读取文件而不被爬虫抓取？
php_value session.cookie_lifetime 18000
php_value session.gc_maxlifetime 18000
RewriteEngine On
RewriteCond %{HTTPS} off
RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI}
<IfModule mod_deflate.c>
# Compress HTML, CSS, JavaScript, Text, XML and fonts
AddOutputFilterByType DEFLATE application/javascript
AddOutputFilterByType DEFLATE application/rss+xml
AddOutputFilterByType DEFLATE application/vnd.ms-fontobject
AddOutputFilterByType DEFLATE application/x-font
AddOutputFilterByType DEFLATE application/x-font-opentype
AddOutputFilterByType DEFLATE application/x-font-otf
AddOutputFilterByType DEFLATE application/x-font-truetype
AddOutputFilterByType DEFLATE application/x-font-ttf
AddOutputFilterByType DEFLATE application/x-javascript
AddOutputFilterByType DEFLATE application/xhtml+xml
AddOutputFilterByType DEFLATE application/xml
AddOutputFilterByType DEFLATE font/opentype
AddOutputFilterByType DEFLATE font/otf
AddOutputFilterByType DEFLATE font/ttf
AddOutputFilterByType DEFLATE image/svg+xml
AddOutputFilterByType DEFLATE image/x-icon
AddOutputFilterByType DEFLATE text/css
AddOutputFilterByType DEFLATE text/html
AddOutputFilterByType DEFLATE text/javascript
AddOutputFilterByType DEFLATE text/plain
AddOutputFilterByType DEFLATE text/xml
# Remove browser bugs (only needed for really old browsers)
BrowserMatch ^Mozilla/4 gzip-only-text/html
BrowserMatch ^Mozilla/4\.0[678] no-gzip
BrowserMatch \bMSIE !no-gzip !gzip-only-text/html
Header append Vary User-Agent
</IfModule>
<IfModule mod_expires.c>
ExpiresActive On
# Images
ExpiresByType image/jpeg "access plus 1 year"
ExpiresByType image/gif "access plus 1 year"
ExpiresByType image/png "access plus 1 year"
ExpiresByType image/webp "access plus 1 year"
ExpiresByType image/svg+xml "access plus 1 year"
ExpiresByType image/x-icon "access plus 1 year"
# Video
ExpiresByType video/webm "access plus 1 year"
ExpiresByType video/mp4 "access plus 1 year"
ExpiresByType video/mpeg "access plus 1 year"
# Fonts
ExpiresByType font/ttf "access plus 1 year"
ExpiresByType font/otf "access plus 1 year"
ExpiresByType font/woff "access plus 1 year"
ExpiresByType font/woff2 "access plus 1 year"
ExpiresByType application/font-woff "access plus 1 year"
# CSS, JavaScript
ExpiresByType text/css "access plus 1 month"
ExpiresByType text/javascript "access plus 1 month"
ExpiresByType application/javascript "access plus 1 month"
# Others
ExpiresByType application/pdf "access plus 1 month"
ExpiresByType image/vnd.microsoft.icon "access plus 1 year"
</IfModule>
# BEGIN cPanel-generated php ini directives, do not edit
# Manual editing of this file may result in unexpected behavior.
# To make changes to this file, use the cPanel MultiPHP INI Editor (Home >> Software >> MultiPHP INI Editor)
# For more information, read our documentation (https://go.cpanel.net/EA4ModifyINI)
<IfModule php7_module>
php_flag display_errors On
php_value max_execution_time 300
php_value max_input_time 600
php_value max_input_vars 10000
php_value memory_limit 320M
php_value post_max_size 80M
php_value session.gc_maxlifetime 14400
php_value session.save_path "/var/cpanel/php/sessions/ea-php74"
php_value upload_max_filesize 20M
php_flag zlib.output_compression Off
</IfModule>
<IfModule lsapi_module>
php_flag display_errors On
php_value max_execution_time 300
php_value max_input_time 600
php_value max_input_vars 10000
php_value memory_limit 320M
php_value post_max_size 80M
php_value session.gc_maxlifetime 14400
php_value session.save_path "/var/cpanel/php/sessions/ea-php74"
php_value upload_max_filesize 20M
php_flag zlib.output_compression Off
</IfModule>
# END cPanel-generated php ini directives, do not edit
# php -- BEGIN cPanel-generated handler, do not edit
# Set the “ea-php74” package as the default “PHP” programming language.
<IfModule mime_module>
AddHandler application/x-httpd-ea-php74___lsphp .php .php7 .phtml
</IfModule>
# php -- END cPanel-generated handler, do not edit
<Files 403.shtml>
order allow,deny
allow from all
</Files>
deny from 45.129.137.249
deny from 43.242.128.31
deny from 104.131.124.162
deny from 216.144.247.254
RewriteRule ^blog=Professional_Packing_and_Wrapping_Services/?$ blogs.php?blog=1 [NC]
RewriteRule ^blog=GIT,_All_Risk_and_General_Insurance/?$ blogs.php?blog=2 [NC]
RewriteRule ^blog=Relocation_Station/?$ blogs.php?blog=4 [NC]
RewriteRule ^category=Residential_Relocation/?$ blogs.php?cat=1 [NC]
RewriteRule ^category=Commercial_Relocation/?$ blogs.php?cat=2 [NC]
RewriteRule ^category=International_Relocation/?$ blogs.php?cat=3 [NC]
RewriteRule ^category=Pet_Transport/?$ blogs.php?cat=4 [NC]
RewriteRule ^category=Car_Transport/?$ blogs.php?cat=5 [NC]
RewriteRule ^category=Wrapping_Services/?$ blogs.php?cat=6 [NC]
RewriteRule ^category=Packing_Services/?$ blogs.php?cat=7 [NC]
RewriteRule ^category=Courier_Services/?$ blogs.php?cat=8 [NC]
RewriteRule ^category=Cleaning_Services/?$ blogs.php?cat=9 [NC]
RewriteRule ^category=Storage/?$ blogs.php?cat=10 [NC]
RewriteRule ^category=Insurance/?$ blogs.php?cat=11 [NC]
RewriteRule ^category=Moving_Tips/?$ blogs.php?cat=12 [NC]
RewriteRule ^category=General/?$ blogs.php?cat=13 [NC]
RewriteRule ^keyword=relocation/?$ blogs.php?key=1 [NC]
RewriteRule ^keyword=residential/?$ blogs.php?key=2 [NC]
RewriteRule ^keyword=commercial/?$ blogs.php?key=3 [NC]
RewriteRule ^keyword=international/?$ blogs.php?key=4 [NC]
RewriteRule ^keyword=storage/?$ blogs.php?key=5 [NC]
RewriteRule ^keyword=car/?$ blogs.php?key=6 [NC]
RewriteRule ^keyword=vehicle/?$ blogs.php?key=7 [NC]
RewriteRule ^keyword=pet/?$ blogs.php?key=8 [NC]
RewriteRule ^keyword=insurance/?$ blogs.php?key=9 [NC]
RewriteRule ^keyword=packers/?$ blogs.php?key=10 [NC]
RewriteRule ^keyword=packing/?$ blogs.php?key=11 [NC]
RewriteRule ^keyword=courier/?$ blogs.php?key=12 [NC]
RewriteRule ^keyword=wrapping/?$ blogs.php?key=13 [NC]
RewriteRule ^keyword=cleaning/?$ blogs.php?key=14 [NC]
RewriteRule ^keyword=long_distance/?$ blogs.php?key=15 [NC]
RewriteRule ^keyword=transport/?$ blogs.php?key=16 [NC]
RewriteRule ^keyword=moving/?$ blogs.php?key=17 [NC]
RewriteRule ^keyword=affordable/?$ blogs.php?key=18 [NC]
RewriteRule ^keyword=professional/?$ blogs.php?key=19 [NC]
RewriteRule ^keyword=cost_effective/?$ blogs.php?key=20 [NC]
RewriteRule ^keyword=tips/?$ blogs.php?key=21 [NC]
要消除 SEO 的重复惩罚，您可以阻止所有指向内部 URI 的 URL，即 /blogs.php?blog=<number>、/blogs.php?cat=<number> 和 /blogs.php?key=<number>。
在 RewriteEngine On 行下方插入一条新规则：
RewriteEngine On
# prohibit direct access to internal URIs
RewriteCond %{THE_REQUEST} /blogs\.php\?(blog|cat|key)=\d+ [NC]
RewriteRule ^ - [F]
# existing rules go below this line
美好的一天。
我们有一个用 vanilla PHP 编码的门户网站，其中有一个博客部分，其中
[mysite.com/blog.php?blog=1] 输出所需文件的内容。
这导致我们的 SEO 专家指出它的格式不适合 SEO URL。
然后我们决定使用 .htaccess 来显示名为 URLs
blog=Residential_Relocation -> blogs.php?blog=1输出[mysite.com/blog.php?blog=Residential_Relocation]
但现在它被视为重复。
我们如何才能只从 blog=1 URL 读取文件而不被爬虫抓取？
php_value session.cookie_lifetime 18000
php_value session.gc_maxlifetime 18000
RewriteEngine On
RewriteCond %{HTTPS} off
RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI}
<IfModule mod_deflate.c>
# Compress HTML, CSS, JavaScript, Text, XML and fonts
AddOutputFilterByType DEFLATE application/javascript
AddOutputFilterByType DEFLATE application/rss+xml
AddOutputFilterByType DEFLATE application/vnd.ms-fontobject
AddOutputFilterByType DEFLATE application/x-font
AddOutputFilterByType DEFLATE application/x-font-opentype
AddOutputFilterByType DEFLATE application/x-font-otf
AddOutputFilterByType DEFLATE application/x-font-truetype
AddOutputFilterByType DEFLATE application/x-font-ttf
AddOutputFilterByType DEFLATE application/x-javascript
AddOutputFilterByType DEFLATE application/xhtml+xml
AddOutputFilterByType DEFLATE application/xml
AddOutputFilterByType DEFLATE font/opentype
AddOutputFilterByType DEFLATE font/otf
AddOutputFilterByType DEFLATE font/ttf
AddOutputFilterByType DEFLATE image/svg+xml
AddOutputFilterByType DEFLATE image/x-icon
AddOutputFilterByType DEFLATE text/css
AddOutputFilterByType DEFLATE text/html
AddOutputFilterByType DEFLATE text/javascript
AddOutputFilterByType DEFLATE text/plain
AddOutputFilterByType DEFLATE text/xml
# Remove browser bugs (only needed for really old browsers)
BrowserMatch ^Mozilla/4 gzip-only-text/html
BrowserMatch ^Mozilla/4\.0[678] no-gzip
BrowserMatch \bMSIE !no-gzip !gzip-only-text/html
Header append Vary User-Agent
</IfModule>
<IfModule mod_expires.c>
ExpiresActive On
# Images
ExpiresByType image/jpeg "access plus 1 year"
ExpiresByType image/gif "access plus 1 year"
ExpiresByType image/png "access plus 1 year"
ExpiresByType image/webp "access plus 1 year"
ExpiresByType image/svg+xml "access plus 1 year"
ExpiresByType image/x-icon "access plus 1 year"
# Video
ExpiresByType video/webm "access plus 1 year"
ExpiresByType video/mp4 "access plus 1 year"
ExpiresByType video/mpeg "access plus 1 year"
# Fonts
ExpiresByType font/ttf "access plus 1 year"
ExpiresByType font/otf "access plus 1 year"
ExpiresByType font/woff "access plus 1 year"
ExpiresByType font/woff2 "access plus 1 year"
ExpiresByType application/font-woff "access plus 1 year"
# CSS, JavaScript
ExpiresByType text/css "access plus 1 month"
ExpiresByType text/javascript "access plus 1 month"
ExpiresByType application/javascript "access plus 1 month"
# Others
ExpiresByType application/pdf "access plus 1 month"
ExpiresByType image/vnd.microsoft.icon "access plus 1 year"
</IfModule>
# BEGIN cPanel-generated php ini directives, do not edit
# Manual editing of this file may result in unexpected behavior.
# To make changes to this file, use the cPanel MultiPHP INI Editor (Home >> Software >> MultiPHP INI Editor)
# For more information, read our documentation (https://go.cpanel.net/EA4ModifyINI)
<IfModule php7_module>
php_flag display_errors On
php_value max_execution_time 300
php_value max_input_time 600
php_value max_input_vars 10000
php_value memory_limit 320M
php_value post_max_size 80M
php_value session.gc_maxlifetime 14400
php_value session.save_path "/var/cpanel/php/sessions/ea-php74"
php_value upload_max_filesize 20M
php_flag zlib.output_compression Off
</IfModule>
<IfModule lsapi_module>
php_flag display_errors On
php_value max_execution_time 300
php_value max_input_time 600
php_value max_input_vars 10000
php_value memory_limit 320M
php_value post_max_size 80M
php_value session.gc_maxlifetime 14400
php_value session.save_path "/var/cpanel/php/sessions/ea-php74"
php_value upload_max_filesize 20M
php_flag zlib.output_compression Off
</IfModule>
# END cPanel-generated php ini directives, do not edit
# php -- BEGIN cPanel-generated handler, do not edit
# Set the “ea-php74” package as the default “PHP” programming language.
<IfModule mime_module>
AddHandler application/x-httpd-ea-php74___lsphp .php .php7 .phtml
</IfModule>
# php -- END cPanel-generated handler, do not edit
<Files 403.shtml>
order allow,deny
allow from all
</Files>
deny from 45.129.137.249
deny from 43.242.128.31
deny from 104.131.124.162
deny from 216.144.247.254
RewriteRule ^blog=Professional_Packing_and_Wrapping_Services/?$ blogs.php?blog=1 [NC]
RewriteRule ^blog=GIT,_All_Risk_and_General_Insurance/?$ blogs.php?blog=2 [NC]
RewriteRule ^blog=Relocation_Station/?$ blogs.php?blog=4 [NC]
RewriteRule ^category=Residential_Relocation/?$ blogs.php?cat=1 [NC]
RewriteRule ^category=Commercial_Relocation/?$ blogs.php?cat=2 [NC]
RewriteRule ^category=International_Relocation/?$ blogs.php?cat=3 [NC]
RewriteRule ^category=Pet_Transport/?$ blogs.php?cat=4 [NC]
RewriteRule ^category=Car_Transport/?$ blogs.php?cat=5 [NC]
RewriteRule ^category=Wrapping_Services/?$ blogs.php?cat=6 [NC]
RewriteRule ^category=Packing_Services/?$ blogs.php?cat=7 [NC]
RewriteRule ^category=Courier_Services/?$ blogs.php?cat=8 [NC]
RewriteRule ^category=Cleaning_Services/?$ blogs.php?cat=9 [NC]
RewriteRule ^category=Storage/?$ blogs.php?cat=10 [NC]
RewriteRule ^category=Insurance/?$ blogs.php?cat=11 [NC]
RewriteRule ^category=Moving_Tips/?$ blogs.php?cat=12 [NC]
RewriteRule ^category=General/?$ blogs.php?cat=13 [NC]
RewriteRule ^keyword=relocation/?$ blogs.php?key=1 [NC]
RewriteRule ^keyword=residential/?$ blogs.php?key=2 [NC]
RewriteRule ^keyword=commercial/?$ blogs.php?key=3 [NC]
RewriteRule ^keyword=international/?$ blogs.php?key=4 [NC]
RewriteRule ^keyword=storage/?$ blogs.php?key=5 [NC]
RewriteRule ^keyword=car/?$ blogs.php?key=6 [NC]
RewriteRule ^keyword=vehicle/?$ blogs.php?key=7 [NC]
RewriteRule ^keyword=pet/?$ blogs.php?key=8 [NC]
RewriteRule ^keyword=insurance/?$ blogs.php?key=9 [NC]
RewriteRule ^keyword=packers/?$ blogs.php?key=10 [NC]
RewriteRule ^keyword=packing/?$ blogs.php?key=11 [NC]
RewriteRule ^keyword=courier/?$ blogs.php?key=12 [NC]
RewriteRule ^keyword=wrapping/?$ blogs.php?key=13 [NC]
RewriteRule ^keyword=cleaning/?$ blogs.php?key=14 [NC]
RewriteRule ^keyword=long_distance/?$ blogs.php?key=15 [NC]
RewriteRule ^keyword=transport/?$ blogs.php?key=16 [NC]
RewriteRule ^keyword=moving/?$ blogs.php?key=17 [NC]
RewriteRule ^keyword=affordable/?$ blogs.php?key=18 [NC]
RewriteRule ^keyword=professional/?$ blogs.php?key=19 [NC]
RewriteRule ^keyword=cost_effective/?$ blogs.php?key=20 [NC]
RewriteRule ^keyword=tips/?$ blogs.php?key=21 [NC]
要消除 SEO 的重复惩罚，您可以阻止所有指向内部 URI 的 URL，即 /blogs.php?blog=<number>、/blogs.php?cat=<number> 和 /blogs.php?key=<number>。
在 RewriteEngine On 行下方插入一条新规则：
RewriteEngine On
# prohibit direct access to internal URIs
RewriteCond %{THE_REQUEST} /blogs\.php\?(blog|cat|key)=\d+ [NC]
RewriteRule ^ - [F]
# existing rules go below this line