Filter Tests

Included in this database are filtering tests conducted by testing labs, organizations, or other researchers. Many tests of filtering effectiveness were conducted by advocacy groups or researchers hired by attorneys in legal cases, or were sponsored by filtering vendors.  These affiliations are noted in the “author” column when appropriate.  Filtering research literature is diverse, and not all researchers agree upon appropriate methodology and sampling techniques for evaluating filters. Most filtering studies attempt to measure effectiveness by replicating user behavior, such as searching for websites that a filter should block, then measuring the degree of “under blocking.”  Some filtering studies also measure “over blocking” by searching for content that should not be blocked by the filter.  University of Michigan Professor Paul Resnick, one of the authors of the regarded Kaiser Family Foundation study, “See No Evil“, has written a helpful article on the topic of filtering research in the journal Communications of the ACM entitled “Calculating Error Rates for Filtering Software.”

Click here for the full table, which is in a pdf

Date Author Title Conclusion
Dec 2008 EU Safer Internet Protecting children on the Net with content filtering The trend lines are moving towards 0% mistakes (no overblocking, no underblocking) meaning that overall the tools have made significant improvements in the filtering accuracy.
July 2008 Australian Communications and Media Authority Closed environment testing of ISP-level internet content filtering Successful blocking (the proportion of illegal and inappropriate content that should have been blocked that was successfully blocked) was between 88% and 97% with most achieving over 92%.  Overblocking (the proportion of content that was blocked that should not have been blocked) was between 1% and 6%, with most falling under 3%.
April 2008 Untangle Deep Throat Fight Club All products performed within a small range, probably within the margin of error. If you are expecting to block all porn, all these products will not be effective, however if you’re looking to control and monitor porn usage as much as possible these products make great candidates.
Sep 2007 Paul Resnick (for NCRL) Expert report of Paul Resnick In this study, I found that less than 1/3 of 1% of patron requests for web pages resulted in incorrect blocks.
Aug 2007 Bennett Haselton (for the ACLU) Report on the Accuracy Rate of FortiGuard Extrapolating the results, the number of wrongly blocked websites would be about 76,000.
May 2006 Philip B.  Stark(For the Dept of Justice) Expert Report of Philip B. Stark, PhD. Of the sexually explicit Wordtracker search results, 1.4 percent to 12.8 percent were not blocked by filters. Filters blocked 2.9 percent to 32.8 percent of the clean Wordtracker results. I estimate that 8.8 percent to 60.2 percent of the sexually explicit websites in the Google and MSN indexes are not blocked by the filters. Of the clean websites cataloged by Google or MSN, I estimate that 0.4 percent to 23.6 percent are blocked by filters.
Jan 2006 Broadband testing(For  Blue Coat) URL Filtering Databases Accuracy Test We found that Blue Coat’s WebFilter categorized significantly more URLs than any of the other databases, and more than twice as many as either Websense or Intersafe.
Jan 2006 Veritest(For Websense) Websense: Web Filtering Effectiveness Study In our testing, we found that that Websense provided the greatest selection of predefined categories, the most accurate and diverse category filtering, and the most accurate security categorization.
Jun 2005 Broadband testing(For Blue Coat) Comparing URL Filtering Databases Blue Coat Web Filter (BCWF),especially with DRTR enabled, came out the clear winner, notching up 11 category firsts along the way.
Mar 2004 Report on the evaluation of the final version ofthe NetProtect Product Compared to the other tools, Surf-mate achieved a good score about flexibility aspects, a medium score about security aspects and the best result in filtering accuracy.
June 2003 Online Policy Group Internet Blocking inPublic Schools Blocking software overblocks and underblocks, that is,the software blocks access to many web pages protected by the First Amendment and does not block many of the web pages that CIPA would likely prohibit.
Mar, 2002 eTesting Labs (For Websense) Corporate Content Filtering Performanceand Effectiveness Testing Websense Enterprise v4.3 generated the best positiveblocking effectiveness compared to SuperScout and SmartFilter.
Dec, 2002 Kaiser Family Foundation See No Evil: How Internet Filters Affect the Search for Health Info “At the least restrictive orintermediate configurations, the filters tested do not block a substantial proportion of general health information sites; however, at the most restrictive configuration, one in four health sites are blocked”
Sept 2001 Australian Broadcasting Company Effectiveness of Internet Filtering Software Products
Oct 2001 Ben Edelman, Berkman Center (for the ACLU) Sites Blocked by Internet Filtering Programs I have concluded that installation in libraries of Internet blocking programsconfigured to block particular categories of Internet content will inevitably block Internetcontent that does not meet the programs’ self-defined category definitions
Oct 2001 Dr. Joseph Janes, Univ. of Washinton (For the ACLU) Expert report of Dr. Joseph Janes Overall, then, 165 of the 699 sites evaluated were found not to be of any value or use in a library context, 23.6% of the total. 60, or 8.6% could not be found, and therefore 474 of thesites, or 67.8% are examples of overblocking on the part of these blocking packages.Based on this analysis, I conclude that the blocking programs tested block a significant amount of content that would be appropriate in a library setting.
Oct 2001 Certus Consulting Group (For the DOJ) Internet Filtering Accuracy Review The estimated error rate for the Cyber-Patrol product was estimated lie within the range of 4.69% to 7.99%.  The estimated error rate for Websense was 5.25% to 11.03%; N2H2 6.92% to 9.36%.
Oct 2001 eTesting Labs (For the DOJ) U.S. Department of Justice: WebContent Filtering Software Comparison Four primary content filtering applications tested correctly blocked an average of approximately 92 percent of objectionable content. Four primary content filtering applications tested incorrectly blocked an average of 4 percent of content not matching the test criteria.
Mar 2001 Consumer Reports “Digital Chaperones for Kids” Filtering software is no substitute for parental supervision. Most of the products we tested failed to block one objectionable site in five.”
Oct 2000 PeaceFire (Submitted to COPA Commission) Study of Average Error Rates for Censorware Programs We conclude that any one of the given products blocks large amounts of innocuous material — and that most of the sites blocked by these products have not been reviewed by staff to ensure that the sites meet the company’s criteria.
Mar 2000 Family Research Council (by David Burt) Dangerous Access 2000 [Tacoma Library] nonsexual sites accounted for 0.07 percent. In other words, 99.93 percent of the time the filter functioned properly. [Cincinnati library] only 2 percent of the blocked sites were nonsexual in nature, and that this accounted for only 0.01 percent of all web accesses-or that 99.99 percent of the time thefilter did not block innocent sites.
Mar 1999 Censor ware Project Censored Internet Access in Utah Schools and Libraries
Sept, 1999 University of Penn. – Christopher Hunter Internet Filter Effectiveness: “Put simply, taken all together, filters failed to block objectionable content25 percent of the time, while on the other hand, they improperly blocked 21percent of benign content.”
Dec 1997 EPIC Faulty Filters Our research showed that a family-friendly search engine typically blocked access to 95-99 percent of the material available on the Internet that might be of interest to young people.

David Burt's website

%d bloggers like this: