Page 2 of 2 FirstFirst 12
Results 11 to 19 of 19
  1. #11
    Newbie
    Join Date
    Mar 2019
    Posts
    3

    Default

    Thanks for making this script, WebFool, but I get the following error when running it.

    Code:
    → ~\OneDrive\Desktop› .\easylist2untangle.ps1
    At C:\Users\mrhan\OneDrive\Desktop\easylist2untangle.ps1:74 char:97
    + ... e a few $filenamejson with $numberoflines lines from $easylistsource"
    +                                                                         ~
    The string is missing the terminator: ".
    At C:\Users\mrhan\OneDrive\Desktop\easylist2untangle.ps1:74 char:98
    + ... e a few $filenamejson with $numberoflines lines from $easylistsource"
    +                                                                          ~
    Missing closing ')' in expression.
        + CategoryInfo          : ParserError: (:) [], ParseException
        + FullyQualifiedErrorId : TerminatorExpectedAtEndOfString
    I'm not familiar with PS scripts. I tried adding a # and ) at the end, but then it gives a different error.

    Code:
    Measure-Object : A positional parameter cannot be found that accepts argument 'now'.
    At C:\Users\mrhan\OneDrive\Desktop\easylist2untangle.ps1:9 char:35
    + $easylistsourcecount=($EasyList | Measure-Object –Line).Lines
    +                                   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        + CategoryInfo          : InvalidArgument: (:) [Measure-Object], ParameterBindingException
        + FullyQualifiedErrorId : PositionalParameterNotFound,Microsoft.PowerShell.Commands.MeasureObjectCommand
    Weirdly, it works if I run it line by line. At least, it generates the files, but I still can't import them. It hangs for a few minutes after clicking save, then all the user added filters are blank.
    Last edited by mrhanman; 02-03-2021 at 10:38 PM.

  2. #12
    Untangle Ninja
    WebFooL's Avatar
    Join Date
    Jan 2009
    Location
    Sweden (Eskilstuna)
    Posts
    5,211

    Default

    Looks like you have some Copy-Paste error and missing a ".

    I'll see if i have the time to commit my changes to git tomorrow and you can download the script from there.

  3. #13
    Master Untangler
    Join Date
    Jul 2018
    Posts
    132

    Default

    Same here. Tried importing but on hitting save it just hangs.....

  4. #14
    Untangle Ninja
    WebFooL's Avatar
    Join Date
    Jan 2009
    Location
    Sweden (Eskilstuna)
    Posts
    5,211

    Default

    What if you modify the 5000 to a lower number?

  5. #15
    Master Untangler
    Join Date
    Jul 2018
    Posts
    132

    Default

    I have changed 5000 in the script to 2000 and am running it now.
    Let's see.

  6. #16
    Master Untangler
    Join Date
    Jul 2018
    Posts
    132

    Default

    Before importing I try to make an export of the current filters. I get a 2 byte file!!!
    I think something in this app is broken.....

  7. #17
    Master Untangler
    Join Date
    Jul 2018
    Posts
    132

    Default

    Tried. Same thing. Freezes upon hitting save.

  8. #18
    Untanglit
    Join Date
    May 2019
    Posts
    15

    Default

    I've be playing around with this for a couple days now. Seems the original scrip gave a JSON format that is different then the one I get if export a list from Untangle Ad Blocker. When Imported the results, they were also marked in Red indicating a "Delete" of the rules so when you click save, it process and you have a blank list.

    I made some changes to the JSON output to match a test example I generated by manually adding one entry then exporting it.
    My edits to the sting and order on lines 13 and 33 has resulted in a better JSON but one small problem persists. When it converts from the CVS to JSON, the values are getting put in "" example:"blocked":"true","flagged":"true". The two values that are true should not have the "" around them. This causes a JSON extension error when you try to save.

    Anyone know how to convert the CVS to JSON with those values not in ""?

    Here's my edited version. I changed the strings on lines 13 and 33 to match the JSON export of my user added test.

    Code:
    ##
    ## Script Create by WebFooL for The Untangle Community
    ##
    $easylistsource = "https://easylist.to/easylist/easylist.txt"
    $Request = Invoke-WebRequest $easylistsource
    $EasyList = $Request.Content
    $filenamejson = "ADImport.json"
    $filenamecsv = "ADImport.csv"
    $easylistsourcecount=($EasyList | Measure-Object –Line).Lines
    $hash = $null
    $counter = 0
    $hash = @'
    blocked,flagged,string,javaClass,name,description,readOnly,id,category,enabled
    
    
    '@
    
    
    write-host "Will now work for a while do not panic!"
    ForEach ($line in $($EasyList -split "`n"))
    {
    #Add Nice Progress bar.. 
    Write-Progress -Activity "Processing Easylist" -CurrentOperation $line -PercentComplete (($counter / $easylistsourcecount) * 100)
        #Remove all Commented lines (They all start with !)
        if($line -clike '!*') {
        #Do Nothing
        } elseif($line -eq "[Adblock Plus 2.0]"){
        #Do Nothing
        }elseif($line -eq "" ){
        #Do Nothing
        }else {
            #Create Untangle JSON
            $hash += "true,true,$line,com.untangle.uvm.app.GenericRule,null,null,null,null,null,true`r`n"
            $counter++
        }    
    }
    #Tempstore as CSV
    $hash | Set-Content -Path $filenamecsv
    
    
    function Split-CsvFile {
        param (
            [string]$sourceCSV,
            [int]$size
        )
    
    
        $exportPath = $sourceCSV.Substring(0, $sourceCSV.LastIndexOf('.'))
        $count = (Import-Csv $sourceCSV).count
        $startrow = 0;
        $counter  = 1;
    
    
        while ($startrow -lt $count) {
            $filename = "$($exportPath)_$($counter)"
            Write-host "Filename:" $filename
            Import-CSV $sourceCSV | select-object -skip $startrow -first $size | Export-CSV "$filename.csv" -NoClobber -Encoding UTF8 -NoTypeInformation
            #Convert to Json
            import-csv "$filename.csv" | ConvertTo-Json -Compress | Set-Content -Path "$filename.json"
            if (Test-Path "$filename.csv") {
              Remove-Item "$filename.csv"
            }
            $startrow += $size
            $counter++
        }
    }
    
    
    Split-CsvFile -sourceCSV $filenamecsv -size 500 
    #import-csv $filenamecsv | ConvertTo-Json -Compress | Set-Content -Path $filenamejson
    #Count lines in the CSV
    $numberoflines = (Import-Csv $filenamecsv | Measure-Object -Property string).Count
    #Write friendly exit message
    Write-Host "Done you now have a few $filenamejson with $numberoflines lines from $easylistsource"

  9. #19
    Untanglit
    Join Date
    May 2019
    Posts
    15

    Default

    Turned out my issues were being caused by MS Excel delimiter separating lines at the commas. I manually ingested Easylist into Excel without the delimiter, scrubbed the ! lines and used the CONCATENATE function to consolidate the few lines Excel still wants to separate. I edited down WebFool's scrip to just break out the CSV file I created into JSON files. Good news is, I don't really have to separate them and was successful at importing 50K lines (5,5583KB of data) with no issue.

Page 2 of 2 FirstFirst 12

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  

SEO by vBSEO 3.6.0 PL2