Page 2 of 3 FirstFirst 123 LastLast
Results 11 to 20 of 26
  1. #11
    Newbie
    Join Date
    Mar 2019
    Posts
    3

    Default

    Thanks for making this script, WebFool, but I get the following error when running it.

    Code:
    → ~\OneDrive\Desktop› .\easylist2untangle.ps1
    At C:\Users\mrhan\OneDrive\Desktop\easylist2untangle.ps1:74 char:97
    + ... e a few $filenamejson with $numberoflines lines from $easylistsource"
    +                                                                         ~
    The string is missing the terminator: ".
    At C:\Users\mrhan\OneDrive\Desktop\easylist2untangle.ps1:74 char:98
    + ... e a few $filenamejson with $numberoflines lines from $easylistsource"
    +                                                                          ~
    Missing closing ')' in expression.
        + CategoryInfo          : ParserError: (:) [], ParseException
        + FullyQualifiedErrorId : TerminatorExpectedAtEndOfString
    I'm not familiar with PS scripts. I tried adding a # and ) at the end, but then it gives a different error.

    Code:
    Measure-Object : A positional parameter cannot be found that accepts argument 'now'.
    At C:\Users\mrhan\OneDrive\Desktop\easylist2untangle.ps1:9 char:35
    + $easylistsourcecount=($EasyList | Measure-Object –Line).Lines
    +                                   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        + CategoryInfo          : InvalidArgument: (:) [Measure-Object], ParameterBindingException
        + FullyQualifiedErrorId : PositionalParameterNotFound,Microsoft.PowerShell.Commands.MeasureObjectCommand
    Weirdly, it works if I run it line by line. At least, it generates the files, but I still can't import them. It hangs for a few minutes after clicking save, then all the user added filters are blank.
    Last edited by mrhanman; 02-03-2021 at 10:38 PM.

  2. #12
    Untangle Ninja
    WebFooL's Avatar
    Join Date
    Jan 2009
    Location
    Sweden (Eskilstuna)
    Posts
    5,245

    Default

    Looks like you have some Copy-Paste error and missing a ".

    I'll see if i have the time to commit my changes to git tomorrow and you can download the script from there.

  3. #13
    Master Untangler
    Join Date
    Jul 2018
    Posts
    139

    Default

    Same here. Tried importing but on hitting save it just hangs.....

  4. #14
    Untangle Ninja
    WebFooL's Avatar
    Join Date
    Jan 2009
    Location
    Sweden (Eskilstuna)
    Posts
    5,245

    Default

    What if you modify the 5000 to a lower number?

  5. #15
    Master Untangler
    Join Date
    Jul 2018
    Posts
    139

    Default

    I have changed 5000 in the script to 2000 and am running it now.
    Let's see.

  6. #16
    Master Untangler
    Join Date
    Jul 2018
    Posts
    139

    Default

    Before importing I try to make an export of the current filters. I get a 2 byte file!!!
    I think something in this app is broken.....

  7. #17
    Master Untangler
    Join Date
    Jul 2018
    Posts
    139

    Default

    Tried. Same thing. Freezes upon hitting save.

  8. #18
    Untanglit
    Join Date
    May 2019
    Posts
    15

    Default

    I've be playing around with this for a couple days now. Seems the original scrip gave a JSON format that is different then the one I get if export a list from Untangle Ad Blocker. When Imported the results, they were also marked in Red indicating a "Delete" of the rules so when you click save, it process and you have a blank list.

    I made some changes to the JSON output to match a test example I generated by manually adding one entry then exporting it.
    My edits to the sting and order on lines 13 and 33 has resulted in a better JSON but one small problem persists. When it converts from the CVS to JSON, the values are getting put in "" example:"blocked":"true","flagged":"true". The two values that are true should not have the "" around them. This causes a JSON extension error when you try to save.

    Anyone know how to convert the CVS to JSON with those values not in ""?

    Here's my edited version. I changed the strings on lines 13 and 33 to match the JSON export of my user added test.

    Code:
    ##
    ## Script Create by WebFooL for The Untangle Community
    ##
    $easylistsource = "https://easylist.to/easylist/easylist.txt"
    $Request = Invoke-WebRequest $easylistsource
    $EasyList = $Request.Content
    $filenamejson = "ADImport.json"
    $filenamecsv = "ADImport.csv"
    $easylistsourcecount=($EasyList | Measure-Object –Line).Lines
    $hash = $null
    $counter = 0
    $hash = @'
    blocked,flagged,string,javaClass,name,description,readOnly,id,category,enabled
    
    
    '@
    
    
    write-host "Will now work for a while do not panic!"
    ForEach ($line in $($EasyList -split "`n"))
    {
    #Add Nice Progress bar.. 
    Write-Progress -Activity "Processing Easylist" -CurrentOperation $line -PercentComplete (($counter / $easylistsourcecount) * 100)
        #Remove all Commented lines (They all start with !)
        if($line -clike '!*') {
        #Do Nothing
        } elseif($line -eq "[Adblock Plus 2.0]"){
        #Do Nothing
        }elseif($line -eq "" ){
        #Do Nothing
        }else {
            #Create Untangle JSON
            $hash += "true,true,$line,com.untangle.uvm.app.GenericRule,null,null,null,null,null,true`r`n"
            $counter++
        }    
    }
    #Tempstore as CSV
    $hash | Set-Content -Path $filenamecsv
    
    
    function Split-CsvFile {
        param (
            [string]$sourceCSV,
            [int]$size
        )
    
    
        $exportPath = $sourceCSV.Substring(0, $sourceCSV.LastIndexOf('.'))
        $count = (Import-Csv $sourceCSV).count
        $startrow = 0;
        $counter  = 1;
    
    
        while ($startrow -lt $count) {
            $filename = "$($exportPath)_$($counter)"
            Write-host "Filename:" $filename
            Import-CSV $sourceCSV | select-object -skip $startrow -first $size | Export-CSV "$filename.csv" -NoClobber -Encoding UTF8 -NoTypeInformation
            #Convert to Json
            import-csv "$filename.csv" | ConvertTo-Json -Compress | Set-Content -Path "$filename.json"
            if (Test-Path "$filename.csv") {
              Remove-Item "$filename.csv"
            }
            $startrow += $size
            $counter++
        }
    }
    
    
    Split-CsvFile -sourceCSV $filenamecsv -size 500 
    #import-csv $filenamecsv | ConvertTo-Json -Compress | Set-Content -Path $filenamejson
    #Count lines in the CSV
    $numberoflines = (Import-Csv $filenamecsv | Measure-Object -Property string).Count
    #Write friendly exit message
    Write-Host "Done you now have a few $filenamejson with $numberoflines lines from $easylistsource"

  9. #19
    Untanglit
    Join Date
    May 2019
    Posts
    15

    Default

    Turned out my issues were being caused by MS Excel delimiter separating lines at the commas. I manually ingested Easylist into Excel without the delimiter, scrubbed the ! lines and used the CONCATENATE function to consolidate the few lines Excel still wants to separate. I edited down WebFool's scrip to just break out the CSV file I created into JSON files. Good news is, I don't really have to separate them and was successful at importing 50K lines (5,5583KB of data) with no issue.

  10. #20
    Newbie
    Join Date
    Nov 2020
    Posts
    12

    Default

    Does untangle even work with easylist formats? My tests with || doesn't seem to block anything. I will have to read up (or get help here) on which formats work with the web filter.

    I wrote a script for adguards normal list which is less intrusive and has less bloat. This could easilly be rewritten for any other list if you know the untangle formatting.

    Regex to format the lines to work with untangle rules (link to source in code, so you can modify it for whatever lists you find).
    Regex match anything to exclude lines.

    https://github.com/returaxel/Untangl...-webfilter.ps1

    It iterates easylist in 1-2 seconds on a decent machine.
    Last edited by ccdmnk; 06-16-2021 at 02:05 AM.

Page 2 of 3 FirstFirst 123 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  

SEO by vBSEO 3.6.0 PL2