PowerShell cmdLet Invoke-WebRequest

Returned object

Invoke-WebRequest returns an object whose type is Microsoft.PowerShell.Commands.BasicHtmlWebResonseObject or Microsoft.PowerShell.Commands.HtmlWebResonseObject.

Both classes derive from Microsoft.PowerShell.Commands.WebResonseObject.

PS C:\> $response = invoke-webRequest https://renenyffenegger.ch
PS C:\> $response.GetType().FullName
Microsoft.PowerShell.Commands.HtmlWebResonseObject

It seems that PowerShell 7 always returns a Microsoft.PowerShell.Commands.BasicHtmlWebResponseObject object.

PS7 C:\> $response = invoke-webRequest https://renenyffenegger.ch
PS7 C:\> $response.GetType().FullName
Microsoft.PowerShell.Commands.BasicHtmlWebResponseObject

Download a file

When invoke-webRequest is invoked with the -outFile parameter, it can be used to download a file:

invoke-webRequest https://foo.xyz/path/to/file.xlsx -outfile $env:temp/file.xlsx

A «private» file can be downloaded from github like so:

$githubToken = 'ghp_…'
$headers = @{ Authorization = "token $githubToken" }
invoke-webRequest -headers @headers https://raw.githubusercontent.com/…

Slow downloads for large files

Using invoke-webRequest turns out to be very slow, especially noticeable when downloading large files, because invoke-webRequest reports every single byte in the progress bar.

This behavior can be turned of by setting the $progressPreference variable to silentlyContinue:

$progressPreference = 'SilentlyContinue'
invoke-webRequest $url -out $outFile

Unfortunatly, invoke-webRequest does not have a command option to set the preference for the duration of the command only.

Alternatively, a file can also be downloaded like so:

(new-object Net.WebClient).DownloadFile($url, $outFile)

-UseDefaultCredential

Specifying -useDefaultCredential seems to use the credentials that were authenticated with Windows authentication (on Windows, of course):

$res = invoke-webRequest -useDefaultCredential 'https://xyz.foo.bar/api/data/v8.0/$metadata'
$res.content > p:\ath\to\file.txt

Error message: The request was aborted: Could not create SSL/TLS secure channel.

The error message The request was aborted: Could not create SSL/TLS secure channel. indicates (sometimes?, always?) that (at least?) TLS 1.2 should be used. TLS 1.2 can be enabled with

[net.servicePointManager]::securityProtocol = [net.securityProtocolType]::Tls12

Checking status, content, headers etc.

Assign the result of a web request to a variable:

[net.servicePointManager]::securityProtocol = [net.securityProtocolType]::Tls12
$res = invoke-webRequest https://renenyffenegger.ch/notes/Windows/PowerShell/command-inventory/noun/webRequest/invoke

Now, this variable can be used to query the request's status code…

'{0} {1}' -f $res.statusCode, $res.statusDescription

… some headers …

$res.headers['Content-Type']
$res.headers['Connection']

… or the requests body …

$res.content

… links …

$res.links | foreach-object { '{0,-60} {1}' -f $_.innerText, $_.href}

With the parsedHtml property, it is even possible to access elements in the returned HTML document:

$res.parsedHtml.title
$res.ParsedHtml.body.childNodes.item(3).innerHtml

Proxies

PS C:\> invoke-webRequest   https://somewhere.xy/file.txt -outFile file.txt
Invoke-WebRequest :
Firewall Authentication
You must authenticate to use this service.
…

Determine address/URL of proxy:

PS C:\> netsh winhttp show proxy
 
Current WinHTTP proxy settings:
 
    Proxy Server(s) :  proxy.foobarbaz.xy:8080
    Bypass List     :  *.foobarbaz.xy;intra.net

Determine my domain and username:

PS C:\> whoami
theWindowsDomain\rene

Create a secure string:

PS C:\> $pw = convertTo-secureString 'theSecretPassword' -asPlainText –force

Create a PSCredential object:

PS C:\> $cred = new-object System.Management.Automation.PSCredential -argumentList "theWindowsDomain\Rene", $pw

Execute the webrequest using the -proxy option. Make sure to prefix the proxy URL obtained using winhttp show… with http://

PS C:\> invoke-webRequest –proxy http://proxy.foobarbaz.xy –proxyCredential $cred https://somewhere.xy/file.txt -outFile file.txt
…

Specifying HTTP Headers

The following example executes a HTTP request with a POST method and specifies the values for the two HTTP headers Accept and Content-Type.

The body of the request contains a JSON document.

invoke-webRequest `

  -useDefaultCredentials `
  -method post `
  -headers  @{ 'Accept'        = 'application/json' ;
               'Content-Type'  = 'application/json' } `
$URL   `

  -body '{
     "num”:  42,
     "txt”: "Hello world"
}'

-UseBasicParsing

Using -useBasicParsing is necessary on systems where IE is not installed or configured (for example after a fresh installation of Windows where Internet Explorer was not launched for a first time, or on a Server Core installation of Windows Server).

On such systems, invoke-webRequest throws the error message

The response content cannot be parsed because the Internet Explorer engine is not available, or Internet Explorer's first-launch configuration is not complete. Specify the UseBasicParsing parameter and try again.

On a non-Windows Server machine, the following modification in the registry allows to use invoke-webRequest without starting Internet Explorer:

set-itemProperty -path "HKCU:\SOFTWARE\Microsoft\Internet Explorer\Main" -Name "DisableFirstRunCustomize" -Value 2

The underlying connection was closed …

Sometimes, trying to execute invoke-webRequest results in the error message The underlying connection was closed: An unexpected error occurred on a send.

In this case, the following statement might help:

[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12

Dealing with unsuccessful requests

-SkipHttpErrorCheck

-skipHttpErrorCheck is a switch-parameter that was added in PowerShell 7. With this flag, the invoke-webRequest won't throw an exception if the HTTP status indicates an unsuccessful request.

Test

#
#   Inspired by https://stackoverflow.com/a/19122582/180275
#

set-strictMode -version latest

function get-url {
   param (
      [string] $url
   )


   if ($psVersionTable.psVersion.major -ge 7) {
      $res = invoke-webRequest -skipHttpErrorCheck $url
   }
   else {

      $res = try {
         invoke-webRequest $url
      }
      catch {
         $_.exception.response
      }
   }

   write-host ""
   write-host "  Status for ${url}: $($res.statusCode)"
   write-host "  type of `$res: $($res.GetType().FullName)"
}


write-host "psVersion: $($psVersionTable.psVersion)"

get-url https://google.com/
get-url https://google.com/does-not-exist

Github repository about-PowerShell, path: /cmdlets/webRequest/invoke/deal-with-unsuccessful-requests.ps1

Setting the user agent (-userAgent)

The parameter -userAgent allows to set the user agent string for a request.

PowerShell provides some templates for user agent strings in the Microsoft.PowerShell.Commands.PSUserAgent class:

invoke-webRequest -userAgent ([Microsoft.PowerShell.Commands.PSUserAgent]::Chrome) https://…/…

Alias

wget (compare with wget.exe) and curl (compare with curl.exe) are an aliases for invoke-webRequest.