If you remove the
ETag header, you will totally eliminate
If-None-Match requests and their
304 Not Modified Responses.
I am often logged in to my servers via SSH, and I need to download a file like a WordPress plugin. I've noticed many sites now employ a means of blocking robots like wget from accessing their files. Most of the time they use .htaccess to do this. So a permanent workaround has wget mimick a normal browser.
Apache Web Server users have problems getting Apache Authentication/password-protection in htaccess working, this is a troubleshooting guide to get Password Protection working!
WordPress blogs show the same duplicate content for
https://www.askapache.com/. If you've read about using a robots.txt file for WordPress SEO, than you already understand this setup results in Duplicate Content penalties being levied against your Blog and Web Site by Search Engines.