Results 1 to 6 of 6
  1. #1
    Newbie
    Join Date
    Mar 2013
    Posts
    10

    Post AdBlock Plus to Ad Blocker filter conversion

    After lots of trial and error, it appears the way that Untangle Ad Blocker works is that it matches a filter rule to the URL of the web resource (page, css, js, image, etc.) Only the wild card * works. Items found in the AdBlock Plus filter list (^, ||, $third-party, etc.) don't work. These elements then should be removed from the AdBlock Plus filter list. I Chose the EasyList No Element Hide because Ad Blocker can't hide elements either.

    To accomplish this, I modified the script by WebFooL to do the following:

    • Downloads latest EasyList No Element Hide list
    • Strips out first line
    • Strips out lines with ! since they are comments
    • Strips out lines with @@ since they are pass rules
    • Remove everything to the right of $ since this is usually third-party or other helpers
    • Remove empty lines


    You can get it at GitHub https://github.com/fsSnowboard/Untan...lter-Converter or copy it below.

    Known Issue: At this time, the output files cause Ad Blocker to block too much, and some root domains. I haven't had time to look into this issue yet.

    Code:
    <?php
    /* ********************************************* */
    /* Original script by WebFooL on Untangle Forums */
    /* Modifications by fsSnowboard */
    /* */
    /* Use at your own risk */
    /* ********************************************* */
    
    
    $remote_filter_list = "https://easylist-downloads.adblockplus.org/easylist_noelemhide.txt";
    $timezone_adjust = 60 * 60 * 7; //Currently set to EST
    
    $Content = "./easylist_noelemhide.txt";
    
    $lines = file($Content);
    
    $last_modified = preg_replace('/! Last modified: /', '', $lines[3]);
    $last_modified_time = strtotime($last_modified);
    echo "File last modified: ". $last_modified ." (".$last_modified_time.")</br>\n";
    
    $date_match_time = time() - (60 * 60 * 24 * 5) + $timezone_adjust;
    echo "Date Match time (only used for matching if last modified is less than current date): ". $download_date_match ."(".$date_match_time.")</br>\n";
    
    $download_match_time = $last_modified_time + (60 * 60 * 24 * 5) + $timezone_adjust;
    $download_date_match = date("d M Y H:i T", $date_match_time);
    
    echo "New download on: ". date("d M Y H:i", $download_match_time) ." (".$download_match_time.")</br>\n";
    
    if($last_modified_time <= $date_match_time) {
    echo "File is 5 days old, downloading new one.";
    //if file is 5 days old, download new one
    $new_easylist = file_get_contents($remote_filter_list);
    file_put_contents("easylist_noelemhide.txt", $new_easylist);
    
    //read new file
    $Content = "./easylist_noelemhide.txt";
    $lines = file($Content);
    }
    
    
    
    $badcharacters = array("#", '"', "'", "[", "]", "^", "\n", "\t", "\r", "||");
    
    
    unset($lines[0]); //Line 0 is [Adblock Plus 2.0] and not needed
    
    
    
    foreach ($lines as $key => $value) {
    //strip out lines begining with ! because they are comments
        if($value{0} == "!") {
         unset($lines[$key]);
        }
    }
    
    //remove line with @@ which are pass rules
    foreach ($lines as $key => $value) {
        if($value{0} == "@" && $value{1} == "@") {
         unset($lines[$key]);
        }
    }
    
    //foreach ($lines as $key => $value) {
    
        //strip everything to the right and including ^
        //$pos = strpos($value, "^");
    //if ($pos !== FALSE) {
    // //echo $key." ".$pos."\n";
    // //echo "part:". substr($value, 0, $pos)."\n";
    // //echo "full:".$value."\n";
        // $lines[$key] = substr($value, 0, $pos);
        //}
    //}
    
    //strip everything to the right of $
    foreach ($lines as $key => $value) {
        $pos = strpos($value, '$');
        if ($pos !== FALSE) {
    $lines[$key] = substr($value, 0, $pos);
    }
    }
    
    //Remove empty lines
    foreach ($lines as $key => $value) {
        if($value == "") {
         unset($lines[$key]);
        }
    }
    
    //strip out bad characters
    foreach ($lines as $key => $value) {
    $lines[$key] = str_replace($badcharacters, "", $value);
    }
    
    $lines = array_values(array_unique($lines)); //repair index
    
    echo "Filter Items: ". count($lines) ."<br />\n";
    
    $linesSplit = array_chunk( $lines, 2000 );
    //print_r($linesSplit);
    
    
    $i = 0;
    foreach ($linesSplit as $inner_array) {
        $i++;
        $fp = fopen('ABimport'.$i.'.json', 'w');
        $filestart = "[";
        fwrite($fp, $filestart);
        while (list($key, $value) = each($inner_array))
        {
            //$cleanstr = str_replace($badcharacters, "", $value);
         $store = '{"enabled":true,"string":"'.$value.'","javaClass":"com.untangle.uvm.node.GenericRule"},';
    fwrite($fp, $store);
        }
        $fileend = "]";
        fwrite($fp, $fileend);
        fclose($fp);
    }
    ?>
    I don't consider myself a programmer, so the code could probably be more efficient, but it works.

  2. #2
    Untangle Ninja
    WebFooL's Avatar
    Join Date
    Jan 2009
    Location
    Sweden (Eskilstuna)
    Posts
    5,049

    Default

    Nice work.

    I will look more closely at the code tomorrow and if I see anything that can be improved I will let you know.

  3. #3
    Newbie
    Join Date
    Feb 2013
    Posts
    1

    Default

    Thank you!
    If this were reddit I'd give you one full year of gold membership, no hesitation...

    It does however block many TLDs. After some testing I have found the main culprits to be the following two lines that block them:
    |http://ax-d.*/jstag^
    |http://ox-*/jstag^

    They will be written as follows in the json file (ABimport4.json):
    |http:
    |https:

    Removing the two entries will fix it.
    Will do some more testing and post if I find more.

  4. #4
    Newbie
    Join Date
    Mar 2013
    Posts
    10

    Default

    Thanks.

    Looks like you found what was doing most of the blocking. I haven't had time. I'll modify the code soon.

  5. #5
    Newbie
    Join Date
    Mar 2013
    Posts
    10

    Default

    I can't edit my original post with the new code, so it is below. Also, you can download from Github.

    Code:
    <?php
    /* ********************************************* */
    /* Original script by WebFooL on Untangle Forums */
    /* Modifications by fsSnowboard                  */
    /*                                               */
    /* Use at your own risk                          */
    /* ********************************************* */
    
    
    $remote_filter_list = "https://easylist-downloads.adblockplus.org/easylist_noelemhide.txt";
    $timezone_adjust = 60 * 60 * 7;  //Currently set to EST
    
    $Content = "./easylist_noelemhide.txt";
    
    $lines = file($Content);
    
    $last_modified =  preg_replace('/! Last modified: /', '', $lines[3]);
    $last_modified_time = strtotime($last_modified);
    echo "File last modified: ". $last_modified ." (".$last_modified_time.")</br>\n";
    
    $date_match_time = time() - (60 * 60 * 24 * 5) + $timezone_adjust;
    echo "Date Match time (only used for matching if last modified is less than current date): ". $download_date_match ."(".$date_match_time.")</br>\n";
    
    $download_match_time = $last_modified_time + (60 * 60 * 24 * 5) + $timezone_adjust;
    $download_date_match = date("d M Y H:i T", $date_match_time);
    
    echo "New download on: ". date("d M Y H:i", $download_match_time) ." (".$download_match_time.")</br>\n";
    
    if($last_modified_time <= $date_match_time) {
    	echo "File is 5 days old, downloading new one.";
    	//if file is 5 days old, download new one
    	$new_easylist = file_get_contents($remote_filter_list);
    	file_put_contents("easylist_noelemhide.txt", $new_easylist);
    	
    	//read new file
    	$Content = "./easylist_noelemhide.txt";  
    	$lines = file($Content);
    }
    
    
    
    $badcharacters = array("#", '"', "'", "[", "]", "^", "\n", "\t", "\r", "||");
    
    
    unset($lines[0]);  //Line 0 is [Adblock Plus 2.0] and not needed
    
    
    
    foreach ($lines as $key => $value) {
    	//strip out lines begining with ! because they are comments
        if($value{0} == "!") {
        	unset($lines[$key]);
        }
    }
    
    //remove line with @@ which are pass rules
    foreach ($lines as $key => $value) {
        if($value{0} == "@" && $value{1} == "@") {
        	unset($lines[$key]);
        }
    }
    
    //foreach ($lines as $key => $value) {
    	
        //strip everything to the right and including ^
        //$pos = strpos($value, "^");
    	//if ($pos !== FALSE) {
    	//    //echo $key." ".$pos."\n";
    	//	//echo "part:". substr($value, 0, $pos)."\n";
    	//	//echo "full:".$value."\n";
        //	$lines[$key] = substr($value, 0, $pos);
        //}
    //}
    
    //strip everything to the right of $
    foreach ($lines as $key => $value) {
        $pos = strpos($value, '$');
        if ($pos !== FALSE) {
    		$lines[$key] = substr($value, 0, $pos);
    	}
    }
    
    //remove line with |http or |https because they are blocking most things.
    //I can probably make a $badlines variable if I need to delete more lines
    foreach ($lines as $key => $value) {
        if($value == "|http:" || $value == "|https:") {
        	unset($lines[$key]);
        }
    }
    
    //Remove empty lines
    foreach ($lines as $key => $value) {
        if($value == "") {
        	unset($lines[$key]);
        }
    }
    
    //strip out bad characters
    foreach ($lines as $key => $value) {
    	$lines[$key] = str_replace($badcharacters, "", $value);
    }
    
    $lines = array_values(array_unique($lines));  //repair index
    
    echo "Filter Items: ". count($lines) ."<br />\n";
    
    $linesSplit = array_chunk( $lines, 2000 );
    //print_r($linesSplit);
    
    
    $i = 0; 
    foreach ($linesSplit as $inner_array) { 
        $i++; 
        $fp = fopen('ABimport'.$i.'.json', 'w');     
        $filestart = "[";  
        fwrite($fp, $filestart); 
        while (list($key, $value) = each($inner_array)) 
        {
            //$cleanstr = str_replace($badcharacters, "", $value);     
        	$store = '{"enabled":true,"string":"'.$value.'","javaClass":"com.untangle.uvm.node.GenericRule"},';
    		fwrite($fp, $store);
        }
        $fileend = "]";  
        fwrite($fp, $fileend); 
        fclose($fp);
    }
    ?>
    Thanks so much for finding the problem for me.

  6. #6
    Newbie
    Join Date
    Feb 2013
    Location
    United States
    Posts
    2

    Default

    I found this thread via Google

    Thanks! That's exactly what I was looking for... I will look into it now.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  

SEO by vBSEO 3.6.0 PL2