Invoke WebRequest - PowerShell

Overview

Invoke-WebRequest is a PowerShell command that enables the retrieval of web content by sending HTTP/HTTPS requests. It provides versatile options for customizing request parameters, headers, and authentication mechanisms, making it suitable for various web scraping and web automation tasks.

Syntax

Invoke-WebRequest [-Uri] <Uri> [-Method] <string> [-Headers] <hashtable>
[-Body] <object> [-ContentType] <string> [-UseDefaultCredentials] <boolean>
[-AllowRedirection] <boolean> [-UserAgent] <string> [-MaximumRedirection] <int>
[-TimeoutSec] <int> [-WebSession] <object> [-Credential] <PSCredential>
[-Certificate] <X509Certificate2> [-Proxy] <WebProxy> [-CertificateFile] <string>
[-Unsafe] <boolean> [-OutputFile] <string> [-ThrottleLimit] <int>
[-ProxyUseDefaultCredentials] <boolean> [-SkipCertCheck] <boolean>
[-Accept] <string> [-Authentication] <string> [-Expect100Continue] <boolean>
[-KeepAlive] <boolean> [-PipelineVariable] <string> [-ErrorAction] <string>
[-WarningAction] <string> [-Verbose] [-Debug] [-InformationAction] <string>

Options/Flags

  • -Uri (Mandatory): Specifies the target URI for the web request.
  • -Method (Mandatory): Sets the HTTP request method (e.g., GET, POST, PUT, DELETE).
  • -Headers (Optional): Allows the addition of custom HTTP headers to the request.
  • -Body (Optional): Provides the payload (e.g., data or JSON) for POST requests.
  • -ContentType (Optional): Specifies the MIME type of the request body.
  • -UseDefaultCredentials (Optional): Indicates whether to use the default system credentials for authentication.
  • -AllowRedirection (Optional): Enables automatic redirection to other URLs.
  • -UserAgent (Optional): Sets the User-Agent header to simulate specific browser behavior.
  • -MaximumRedirection (Optional): Limits the number of redirects allowed before the request fails.
  • -TimeoutSec (Optional): Specifies the number of seconds to wait for a response before timing out.
  • -WebSession (Optional): Allows reuse of an existing web session for persistent connections.
  • -Credential (Optional): Provides custom credentials for the request.
  • -Certificate (Optional): Uses a specific X.509 certificate for TLS/SSL authentication.
  • -Proxy (Optional): Sets a proxy server for the request.
  • -CertificateFile (Optional): Alternative method to specify the certificate file path.
  • -Unsafe (Optional): Bypasses SSL/TLS security checks (use with caution).
  • -OutputFile (Optional): Saves the response content to a specified file.
  • -ThrottleLimit (Optional): Enforces a limit on concurrent requests to prevent overloading.
  • -ProxyUseDefaultCredentials (Optional): Indicates whether to use default credentials for proxy authentication.
  • -SkipCertCheck (Optional): Bypasses TLS/SSL certificate validation (not recommended for secure operations).
  • -Accept (Optional): Specifies the acceptable MIME types for the response.
  • -Authentication (Optional): Sets the authentication method for the request (e.g., Basic, Negotiate, NTLM).
  • -Expect100Continue (Optional): Indicates that the client expects a 100-Continue response from the server.
  • -KeepAlive (Optional): Enables keep-alive connections for subsequent requests.
  • -PipelineVariable (Optional): Outputs response data as a PowerShell variable.
  • -ErrorAction, -WarningAction, -InformationAction (Optional): Controls how errors, warnings, and informational messages are handled.
  • -Verbose, -Debug (Optional): Enable verbose or debug output for troubleshooting.

Examples

# Get HTML content from a website
$html = Invoke-WebRequest -Uri "https://example.com"

# POST data to a web service
$body = @{name="John"; age=30}
Invoke-WebRequest -Uri "https://api.example.com/register" -Method "POST" -Body $body

# Save a file from a URL
Invoke-WebRequest -Uri "https://files.example.com/file.zip" -OutputFile "./file.zip"

Common Issues

  • 404 Not Found: The specified URI may not exist or be accessible.
  • Timeout: The request may have exceeded the specified timeout period.
  • SSL/TLS Errors: Certificate issues or insecure connections may cause validation failures.
  • Authentication Failed: Incorrect or insufficient credentials for the request.
  • Access Denied: The server may restrict access to the requested resource.

Integration

Invoke-WebRequest can be integrated with other PowerShell commands or tools for advanced scenarios:

  • Use Select-Object to parse HTML content and extract specific elements.
  • Convert the response content to JSON or XML using ConvertFrom-Json or ConvertFrom-Xml.
  • Automate web scraping tasks by scheduling the command with Invoke-Command.