AdBlock Plus to Ad Blocker filter conversion

[fsSnowboard]

After lots of trial and error, it appears the way that Untangle Ad Blocker works is that it matches a filter rule to the URL of the web resource (page, css, js, image, etc.) Only the wild card * works. Items found in the AdBlock Plus filter list (^, ||, $third-party, etc.) don't work. These elements then should be removed from the AdBlock Plus filter list. I Chose the EasyList No Element Hide because Ad Blocker can't hide elements either.

To accomplish this, I modified the script by WebFooL to do the following:

• Downloads latest EasyList No Element Hide list
• Strips out first line
• Strips out lines with ! since they are comments
• Strips out lines with @@ since they are pass rules
• Remove everything to the right of $ since this is usually third-party or other helpers
• Remove empty lines

You can get it at GitHub https://github.com/fsSnowboard/Untan...lter-Converter or copy it below.

Known Issue: At this time, the output files cause Ad Blocker to block too much, and some root domains. I haven't had time to look into this issue yet.

<?php
/* ********************************************* */
/* Original script by WebFooL on Untangle Forums */
/* Modifications by fsSnowboard */
/* */
/* Use at your own risk */
/* ********************************************* */


$remote_filter_list = "https://easylist-downloads.adblockplus.org/easylist_noelemhide.txt";
$timezone_adjust = 60 * 60 * 7; //Currently set to EST

$Content = "./easylist_noelemhide.txt";

$lines = file($Content);

$last_modified = preg_replace('/! Last modified: /', '', $lines[3]);
$last_modified_time = strtotime($last_modified);
echo "File last modified: ". $last_modified ." (".$last_modified_time.")</br>\n";

$date_match_time = time() - (60 * 60 * 24 * 5) + $timezone_adjust;
echo "Date Match time (only used for matching if last modified is less than current date): ". $download_date_match ."(".$date_match_time.")</br>\n";

$download_match_time = $last_modified_time + (60 * 60 * 24 * 5) + $timezone_adjust;
$download_date_match = date("d M Y H:i T", $date_match_time);

echo "New download on: ". date("d M Y H:i", $download_match_time) ." (".$download_match_time.")</br>\n";

if($last_modified_time <= $date_match_time) {
echo "File is 5 days old, downloading new one.";
//if file is 5 days old, download new one
$new_easylist = file_get_contents($remote_filter_list);
file_put_contents("easylist_noelemhide.txt", $new_easylist);

//read new file
$Content = "./easylist_noelemhide.txt";
$lines = file($Content);
}



$badcharacters = array("#", '"', "'", "[", "]", "^", "\n", "\t", "\r", "||");


unset($lines[0]); //Line 0 is [Adblock Plus 2.0] and not needed



foreach ($lines as $key => $value) {
//strip out lines begining with ! because they are comments
    if($value{0} == "!") {
     unset($lines[$key]);
    }
}

//remove line with @@ which are pass rules
foreach ($lines as $key => $value) {
    if($value{0} == "@" && $value{1} == "@") {
     unset($lines[$key]);
    }
}

//foreach ($lines as $key => $value) {

    //strip everything to the right and including ^
    //$pos = strpos($value, "^");
//if ($pos !== FALSE) {
// //echo $key." ".$pos."\n";
// //echo "part:". substr($value, 0, $pos)."\n";
// //echo "full:".$value."\n";
    // $lines[$key] = substr($value, 0, $pos);
    //}
//}

//strip everything to the right of $
foreach ($lines as $key => $value) {
    $pos = strpos($value, '$');
    if ($pos !== FALSE) {
$lines[$key] = substr($value, 0, $pos);
}
}

//Remove empty lines
foreach ($lines as $key => $value) {
    if($value == "") {
     unset($lines[$key]);
    }
}

//strip out bad characters
foreach ($lines as $key => $value) {
$lines[$key] = str_replace($badcharacters, "", $value);
}

$lines = array_values(array_unique($lines)); //repair index

echo "Filter Items: ". count($lines) ."<br />\n";

$linesSplit = array_chunk( $lines, 2000 );
//print_r($linesSplit);


$i = 0;
foreach ($linesSplit as $inner_array) {
    $i++;
    $fp = fopen('ABimport'.$i.'.json', 'w');
    $filestart = "[";
    fwrite($fp, $filestart);
    while (list($key, $value) = each($inner_array))
    {
        //$cleanstr = str_replace($badcharacters, "", $value);
     $store = '{"enabled":true,"string":"'.$value.'","javaClass":"com.untangle.uvm.node.GenericRule"},';
fwrite($fp, $store);
    }
    $fileend = "]";
    fwrite($fp, $fileend);
    fclose($fp);
}
?>

I don't consider myself a programmer, so the code could probably be more efficient, but it works.

[WebFooL]

Nice work.

I will look more closely at the code tomorrow and if I see anything that can be improved I will let you know.

[ngcat]

Thank you!
If this were reddit I'd give you one full year of gold membership, no hesitation...

It does however block many TLDs. After some testing I have found the main culprits to be the following two lines that block them:
|http://ax-d.*/jstag^
|http://ox-*/jstag^

They will be written as follows in the json file (ABimport4.json):
|http:
|https:

Removing the two entries will fix it.
Will do some more testing and post if I find more.

[fsSnowboard]

Thanks.

Looks like you found what was doing most of the blocking. I haven't had time. I'll modify the code soon.

[fsSnowboard]

I can't edit my original post with the new code, so it is below. Also, you can download from Github.

<?php
/* ********************************************* */
/* Original script by WebFooL on Untangle Forums */
/* Modifications by fsSnowboard                  */
/*                                               */
/* Use at your own risk                          */
/* ********************************************* */


$remote_filter_list = "https://easylist-downloads.adblockplus.org/easylist_noelemhide.txt";
$timezone_adjust = 60 * 60 * 7;  //Currently set to EST

$Content = "./easylist_noelemhide.txt";

$lines = file($Content);

$last_modified =  preg_replace('/! Last modified: /', '', $lines[3]);
$last_modified_time = strtotime($last_modified);
echo "File last modified: ". $last_modified ." (".$last_modified_time.")</br>\n";

$date_match_time = time() - (60 * 60 * 24 * 5) + $timezone_adjust;
echo "Date Match time (only used for matching if last modified is less than current date): ". $download_date_match ."(".$date_match_time.")</br>\n";

$download_match_time = $last_modified_time + (60 * 60 * 24 * 5) + $timezone_adjust;
$download_date_match = date("d M Y H:i T", $date_match_time);

echo "New download on: ". date("d M Y H:i", $download_match_time) ." (".$download_match_time.")</br>\n";

if($last_modified_time <= $date_match_time) {
	echo "File is 5 days old, downloading new one.";
	//if file is 5 days old, download new one
	$new_easylist = file_get_contents($remote_filter_list);
	file_put_contents("easylist_noelemhide.txt", $new_easylist);
	
	//read new file
	$Content = "./easylist_noelemhide.txt";  
	$lines = file($Content);
}



$badcharacters = array("#", '"', "'", "[", "]", "^", "\n", "\t", "\r", "||");


unset($lines[0]);  //Line 0 is [Adblock Plus 2.0] and not needed



foreach ($lines as $key => $value) {
	//strip out lines begining with ! because they are comments
    if($value{0} == "!") {
    	unset($lines[$key]);
    }
}

//remove line with @@ which are pass rules
foreach ($lines as $key => $value) {
    if($value{0} == "@" && $value{1} == "@") {
    	unset($lines[$key]);
    }
}

//foreach ($lines as $key => $value) {
	
    //strip everything to the right and including ^
    //$pos = strpos($value, "^");
	//if ($pos !== FALSE) {
	//    //echo $key." ".$pos."\n";
	//	//echo "part:". substr($value, 0, $pos)."\n";
	//	//echo "full:".$value."\n";
    //	$lines[$key] = substr($value, 0, $pos);
    //}
//}

//strip everything to the right of $
foreach ($lines as $key => $value) {
    $pos = strpos($value, '$');
    if ($pos !== FALSE) {
		$lines[$key] = substr($value, 0, $pos);
	}
}

//remove line with |http or |https because they are blocking most things.
//I can probably make a $badlines variable if I need to delete more lines
foreach ($lines as $key => $value) {
    if($value == "|http:" || $value == "|https:") {
    	unset($lines[$key]);
    }
}

//Remove empty lines
foreach ($lines as $key => $value) {
    if($value == "") {
    	unset($lines[$key]);
    }
}

//strip out bad characters
foreach ($lines as $key => $value) {
	$lines[$key] = str_replace($badcharacters, "", $value);
}

$lines = array_values(array_unique($lines));  //repair index

echo "Filter Items: ". count($lines) ."<br />\n";

$linesSplit = array_chunk( $lines, 2000 );
//print_r($linesSplit);


$i = 0; 
foreach ($linesSplit as $inner_array) { 
    $i++; 
    $fp = fopen('ABimport'.$i.'.json', 'w');     
    $filestart = "[";  
    fwrite($fp, $filestart); 
    while (list($key, $value) = each($inner_array)) 
    {
        //$cleanstr = str_replace($badcharacters, "", $value);     
    	$store = '{"enabled":true,"string":"'.$value.'","javaClass":"com.untangle.uvm.node.GenericRule"},';
		fwrite($fp, $store);
    }
    $fileend = "]";  
    fwrite($fp, $fileend); 
    fclose($fp);
}
?>

Thanks so much for finding the problem for me.

[JimmyTrager]

I found this thread via Google

Thanks! That's exactly what I was looking for... I will look into it now.