Работа с элементами
В Windows PowerShell есть командлеты, которые умеют работать с элементами, под элементами здесь можно понимать: файлы, папки, ключи реестра и так далее.
- Clear-Item — очищает содержимое элемента, но не удаляет сам элемент;
- Copy-Item – копирует элемент;
- Get-Item — получает элемент в указанном месте;
- Invoke-Item — выполняет действие по умолчанию над указанным элементом;
- Move-Item – перемещает элемент;
- New-Item – создает новый элемент;
- Remove-Item – удаляет указанные элементы;
- Rename-Item — переименовывает элемент в пространстве имен поставщика Windows PowerShell;
- Set-Item — изменяет элемент;
- Get-ChildItem — возвращает элементы и дочерние элементы в одном или нескольких определенных местах;
- Get-Location – выводит информацию о текущем местонахождении.
9.8.4. CodingEventsAPI Examples¶
Let’s test this out with our Coding Events API. To keep things simple, let’s use the branch so we don’t need to worry about setting up a database, a secrets manager, or AADB2C.
Run this branch to start the Coding Events API on your local machine.
9.8.4.1. Example
To get a collection of coding events you could use:
> Invoke-RestMethod -Uri "http://localhost:5000/api/events"
To get an individual coding event entity you could use:
> $CodingEventId = 1 > Invoke-RestMethod -Uri "http://localhost:5000/api/events/$CodingEventId"
9.8.4.2. Example
To delete an existing coding event entity you could use:
> $CodingEventId = 1 > $uri = "http://localhost:5000/api/events/$CodingEventId" > Invoke-RestMethod -Method "Delete" -Uri $uri
Получение содержимого страницы и выбор нужных ссылок
Invoke-WebRequest умеет отправлять запросы HTTP/HTTPS/FTP, парсить ответ и возвращать наборы элементов HTML – ссылки, формы, изображения и т.д. Попробуйте любой сайт так:
Invoke-WebRequest -Uri "http://smartfiction.ru"
Для каждой ссылки легко выводится набор атрибутов.
$Site = "http://smartfiction.ru/" $HttpContent = Invoke-WebRequest -Uri $Site $HttpContent.Links innerHTML : smartfiction innerText : smartfiction outerHTML : <A title=smartfiction href="http://smartfiction.ru/" rel=home>smartfiction</A> outerText : smartfiction tagName : A title : smartfiction href : http://smartfiction.ru/ rel : home
Передав запрос по конвейеру командлету Where-Object, можно получить список всех ссылок на книги в формате mobi (ниже показана только первая).
$HttpContent.Links | Where-Object {$_.innertext -eq "mobi"} | fl innerText, href
innerText : mobi
href : http://convert.smartfiction.ru/?uri=http%3A%2F%2Fsmartfiction.ru%2Fprose%2Fhot_and_cold_blood%2F&format
=mobi
Invoke-RestMethod
Invoke-RestMethod is basically a wrapper cmdlet around Invoke-WebRequest. Invoke-RestMethod does some automatic conversion for you. If the API you are consuming returns JSON then Invoke-RestMethod will return a PowerShell Object which is a result of JSON conversion.
As you can see the $response variable is a PSObject you can start using right away, no need for a manual conversion.
Unfortunately the Status Code and Headers are missing, most times this is ok. It’s a standard that 200 is Ok, 201 is Created, 400 causes an error etc. It’s almost safe to assume when your command works and returns an object that all is ok. I only say almost because not everyone adheres to standards and there may be some off the wall edge cases. Headers are important because some APIs provide ETags to help with caching, a Pages header to tell how many pages of objects there are, or a more common one is API versioning/obsolete flags.
Can I Access an HTML Page With Invoke-RestMethod?
Actually…you can! Invoke-RestMethod doesn’t do any conversion for the HTML content, just returns a string but this could still be useful depending on your use case.
Extracting and monitoring web content with PowerShell
Published March 30, 2017 by
FoxDeploy
This kind of request comes up all the time on StackOverflow and /r/PowerShell.
And it’s an interesting problem to solve. However, nothing motivates like greed, and I recently revisited this topic in order to help me track down the newest must-have item, the Switch.
In fact, this post COULD have been called ‘Finding a Nintendo Switch with PowerShell’!
I have been REALLY wanting a Nintendo Switch, and since I’ll be flying up to NYC next month for Tome’s NYC TechStravaganza (come see me if you’ll be in Manhattan that day!), it’s the perfect justification for She-Who-Holds-The-Wallet for me to get one!
But EVERYWHERE is sold out. Still! 
However, the stores have been receiving inventory every now and then, and I know that when GameStop has it in stock, I want to buy it from them! With that in mind, I knew I just needed a way to monitor the page and alert me when some text on it changes.
Web scraping, here we go!
When to Use Invoke-RestMethod Over Invoke-WebRequest
My personal opinion is never, but I’m a control freak. I like having all the information Invoke-WebRequest provides, but telling you it’s the best way would not be honest. The best way is the way that fits the requirements of your script.
For example, my colleague Jordan Benzing recently pointed me at the API provided by ICanHazDadJoke. This is a perfect use case for Invoke-RestMethod!
Looking at the API Documentation you can see it’s a very straight forward API, no response headers to worry about, no odd ball status codes to take in to account. I would use Invoke-RestMethod for this like so
Easy peasy, no fuss! This is the type of API Invoke-RestMethod is great for.
The Microsoft Graph API, however, is not something I would use Invoke-RestMethod for. I would use Invoke-WebRequest for any endpoint I consume.
I hope this helps clear up some of the confusion about when to use Invoke-WebRequest or Invoke-RestMethod. Invoke-RestMethod is perfect for quick APIs that have no special response information such as Headers or Status Codes, whereas Invoke-WebRequest gives you full access to the Response object and all the details it provides.
Catching exceptions
Combining all the options from above can lead to errors, so let’s see how we can catch these exceptions.
Something like the following will give an error (invalid option to ):
However, this will also give an error (syntactically correct command, but server returns a 404):
You can catch all kinds of exceptions by wrapping the request into a try-catch block:
This will catch all exceptions. If you want to handle certain exceptions differently, use multiple catch statements.
We can access the exception though the pipeline variable . The server response object (obviously only if it is a WebException and not something like a Command exception, ParameterBindException, etc.) can then be accessed via .
How does Invoke-WebRequest command work in PowerShell (invoke-webrequest PowerShell)?
The Invoke-WebRequest cmdlet sends HTTP and HTTPS requests to a web page, web service, or Web API. It parses the response and returns collections of links or a single link (based on the request being sent), images, and other HTML elements.
This cmdlet was introduced in PowerShell 3.0.
Beginning in PowerShell 7.0, Invoke-WebRequest supports proxy configuration defined by environment variables.
Example:
$Response = Invoke-WebRequest -URI https://www.bing.com/search?q=how+many+feet+in+a+mile
$Response.InputFields | Where-Object {
$_.name -like "* Value*"
} | Select-Object Name, Value
name value
---- -----
From Value 1
To Value 5280
The above uses the Invoke-WebRequest cmdlet to send a web request to the Bing.com site.
#################PowerShell script to download file from an external link#####################################
cls
$fileName = "Download_File_Report"
#'yyyyMMddhhmm yyyyMMdd
$enddate = (Get-Date).tostring("yyyyMMddhhmmss")
$logFileName = $fileName +"_"+ $enddate+"_Log.txt"
$invocation = (Get-Variable MyInvocation).Value
$directoryPath = Split-Path $invocation.MyCommand.Path
$directoryPathForLog=$directoryPath+"\"+"LogFiles"
if(!(Test-Path -path $directoryPathForLog))
{
New-Item -ItemType directory -Path $directoryPathForLog
#Write-Host "Please Provide Proper Log Path" -ForegroundColor Red
}
#$logPath = $directoryPath + "\" + $logFileName
$logPath = $directoryPathForLog + "\" + $logFileName
$isLogFileCreated = $False
function Write-Log($logMsg)
{
if(!$isLogFileCreated){
Write-Host "Creating Log File..."
if(!(Test-Path -path $directoryPath))
{
Write-Host "Please Provide Proper Log Path" -ForegroundColor Red
}
else
{
$script:isLogFileCreated = $True
Write-Host "Log File ($logFileName) Created..."
$logMessage = ::Format(" - {0}", $logMsg)
Add-Content -Path $logPath -Value $logMessage
}
}
else
{
$logMessage = ::Format(" - {0}", $logMsg)
Add-Content -Path $logPath -Value $logMessage
}
}
try
{
$folderWhereToDownload = "C:\temp"
$externalFileURL= "https://ExternalLinkURLToDownload/fileName123.xlsx"
$request = ::Create($externalFileURL)
$request.Method = "HEAD"
$response = $request.GetResponse()
$fUri = $response.ResponseUri
$fileName = ::GetFileName($fUri.LocalPath);
$response.Close()
$targetLocation = join-path $folderWhereToDownload $filename
Invoke-WebRequest -Uri $externalFileURL -OutFile $targetLocation
Write-Host "The file successfully has been downloaded" -ForegroundColor Green
}
Catch
{
$ErrorMessage = $_.Exception.Message +"in downloading file from an external link!:"
Write-Host $ErrorMessage -BackgroundColor Red
Write-Log $ErrorMessage
}
#################PowerShell script to download file from an external link - ends here##########################
Sending data and setting the content type
To give our request a body, we can either use the option, the option or use a pipeline. For these examples we will do a POST request, so use .
Before actually sending data, let’s talk about the content type. If you do a POST request, but neither specify a Content-Type header nor use the option, Invoke-WebRequest will automatically set the content type to .
More gotchas: when you do set a Content-Type header via , say and then pipe a utf8 file to like so…
… you may not actually send what you expect, as will not read the piped data as utf8.
Depending on the encoding, you may send something like this:
As something like this (ISO-8859-1):
Instead of like this (UTF-8):
To be on the safe side, make sure to either use the option (and specify the ) and/or use a pipeline, but set the option (instead of the Content-Type header in the ) the Invoke-WebRequest cmdlet provides:
… will properly send the UTF-8 data.
Using -Body
If you want to build your body manually in the command, you can use the option:
For posting form data, you can use a hash table (going with the default here):
Examples
Example 1: Use a stateful web service
This example shows how to use the
cmdlet with a stateful web service.
The first call to
sends a sign-in request. The command specifies a value of «Session» for the value of the
-SessionVariable
parameter, and saves the result in the
variable. When the command completes, the
variable contains an
and the
variable contains a
object. This logs the user into the site.
The call to
by itself shows the
object in the variable.
The second call to
fetches the user’s profile which requires that the user be logged into the site. The session data stored in the
variable is used to provide session cookies to the site created during the login. The result is saved in the
variable.
The call to
by itself shows the
in the variable.
Example 2: Get links from a web page
This command gets the links in a web page.
It uses the
cmdlet to get the web page content.
Then it users the
Links
property of the
that
returns, and the Href property of each link.
Example 3: Writes the response content to a file using the encoding defined in the requested page.
This command uses the
cmdlet to retrieve the web page content of an msdn page.
The first command retrieves the page and saves the response object in a variable.
The second command creates a
to use to write the response content to a file. The
Encoding
property of the response object is used to set the encoding for the file.
The final few commands write the
Content
property to the file then disposes the
.
Note that the
Encoding
property will be null if the web request does not return text content.
Example 4: Submit a multipart/form-data file
This example uses the
cmdlet upload a file as a
submission. The file
will be submitted as the form field
with the
of
.
Парсинг заголовков
В ответ на запрос о ссылке на книгу сервер выдает такую картину.
Invoke-WebRequest -Uri "http://convert.smartfiction.ru/?uri=http%3A%2F%2Fsmartfiction.ru%2Fprose%2Fhot_and_cold_blood%2F&format=mobi"
StatusCode : 200
StatusDescription : OK
Content : {76, 105, 111, 100...}
RawContent : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Connection: keep-alive
Status: 200 OK
content-disposition: attachment; filename="hot_and_cold_blood.mobi"
content-transfer-encoding: binary
x-ua-compat...
Headers : {, , , ...}
RawContentLength : 46350
Имя файла тут есть: filename=»hot_and_cold_blood.mobi». Но я сразу приуныл, потому что извлечь его можно только регулярным выражением, которые я исторически не осилил. Однако меня быстро утешил в Телеграме Вадимс Поданс 
$r = Invoke-WebRequest -Uri "http://convert.smartfiction.ru/?uri=http%3A%2F%2Fsmartfiction.ru%2Fprose%2Fhot_and_cold_blood%2F&format=mobi" $r.Headers -match 'filename=\"(.+)\"' | Out-Null $matches hot_and_cold_blood.mobi
Регулярное выражение берет из ответа заголовки (Headers) и вытаскивает имя файла из секции content-deposition.
В результате получается такой скрипт.
$Site = "http://smartfiction.ru/"
$HttpContent = Invoke-WebRequest -Uri $Site
$HttpContent.Links | Where-Object {$_.innertext -eq "mobi"} |
%{
(Invoke-WebRequest -Uri $_.href).Headers -match 'filename=\"(.+)\"' | Out-Null
Invoke-WebRequest -Uri $_.href -OutFile $matches
}
Invoke-Webrequest
This cmdlet is probably the most common way to use Powershell for retrieving data from a website:
$res = Invoke-WebRequest -Uri 'http://www.example.com' $res = Invoke-WebRequest -Uri 'http://www.example.com' -Outfile index.html $res = Invoke-WebRequest -Uri 'http://www.example.com' -Method POST -Body "postdata"
In contrast to the other methods described in this article, automatically sets the User Agent header. It is possible to influence this using the parameter . In addition, the parameter exists. However, it does not allow customizing certain headers such as User-Agent. Full control over the headers can only be achieved by .NET classes as described below.
If no error occurrs, an object of the type is returned. Depending on the version of Powershell, the properties of this object differ: In version 5, the content supplied by the web server is parsed and is available with the property . This property is missing under Powershell 7.
The automatic parsing can be very useful for quick web scraping tasks. For example, all links of a web page can be extracted with one simple command:
(Invoke-WebRequest -Uri 'https://www.example.com').Links.Href
The entire body of the downloaded page is stored in the property of the result object. If the headers are also desired, they can be retrieved from the property.
If an error occurs, no object is returned. However, the error message contained in the exception can be retrieved:
try
{
$res = Invoke-WebRequest -Uri 'http://getstatuscode.com/401'
}
catch
{
$ex = $_.Exception
}
$msg = $ex.Message
$status = $ex.Response.StatusCode.value__
In case of errors where the web server does return content, for example a custom 401 error page, this content is also accessible via the exception as follows (access to the response headers is also shown):
try
{
$res = Invoke-WebRequest -Uri 'http://getstatuscode.com/401'
}
catch
{
$rs = $_.Exception.Response.GetResponseStream()
$reader = New-Object System.IO.StreamReader($rs)
$content = $reader.ReadToEnd()
foreach($header in $rs.Response.Headers.AllKeys)
{
write-host $('{0}: {1}' -f $header, $rs.Response.Headers)
}
}
When sending multiple requests, e.g. automated in a loop, it sometimes becomes noticeable that can be very slow. This is due to the extensive parsing of the response that is enabled by default. Additionally, a progress bar is often displayed for each request.
The following parameters and settings can help to accelerate:
- this does a little less in-depth processing of the server response, but the , , , and properties are still populated.
- With the following setting the progress bar can be suppressed (see also this Microsoft article).
By default, automatically follows up to 5 redirects. This can be influenced with the paratemeter .
Proxy Support
uses the proxy defined in the Windows settings by default. This can be overridden with the parameter, which takes an URI as argument. If the proxy requires authentication, the credentials can be specified with the parameter, requiring an argument of type PSCredential. This would look something like this:
$secPw = ConvertTo-SecureString '************' -AsPlainText -Force $creds = New-Object System.Management.Automation.PSCredential -ArgumentList 'username', $secPw $res = Invoke-WebRequest -Uri 'https://www.example.com' -Proxy 'http://127.0.0.1:8080' -ProxyCredential $creds
Alternatively, the parameter can be specified which results in using the credentials of the current user.
Sessions and Cookies
is able to use cookies and thus also supports cookie based sessions by specifying the parameter. The corresponding cookies / session can then be used in subsequent requests with the parameter.
Invoke-WebRequest -SessionVariable -Uri 'https://www.google.com' Invoke-WebRequest -WebSession $Session -Uri 'https://www.google.com'
The cookies are available with the property and it is possible to define custom cookies. In the following example, the session object is created in advance, which can be useful to set custom cookies before executing the first request:
$cookie = New-Object System.Net.Cookie $cookie.Name = "specialCookie" $cookie.Value = "value" $cookie.Domain = "domain" $session = New-Object Microsoft.PowerShell.Commands.WebRequestSession $session.Cookies.Add($cookie) Invoke-WebRequest -WebSession $session -Uri 'https://www.example.com'
Introduction to Invoke-WebRequest
Invoke-WebRequest is an interesting PowerShell 3.0 cmdlet; it enables us to extract useful information from web pages.
List the Links on a Web Page
Let us see Invoke-WebRequest in action, this is how to display a list of links on a web page.
# PowerShell Invoke-WebRequest Example$Site = «https://www.computerperformance.co.uk» $Test = Invoke-WebRequest -URI $Site$Test.Links | Foreach {$_.href }
Note 1: I chose my own website, please change the value of $Site for the url of your choice.
Research Properties of Invoke-WebRequest
Here is one of my favorite PowerShell techniques, pipe an object into Get-Member and thus expose a list of the methods and properties.
Clear-Host$Site = «https://www.computerperformance.co.uk» $Test = Invoke-WebRequest $Site$Test | Get-Member
Name MemberType——- ———-Equals MethodGetHashCode MethodGetType MethodToString MethodAllElements PropertyBaseResponse Property Content PropertyForms PropertyHeaders PropertyImages PropertyInputFields PropertyLinks Property ParsedHtml PropertyRawContent PropertyRawContentLength PropertyRawContentStream PropertyScripts Property StatusCode PropertyStatusDescription Property
Note 2: Interesting properties include AllElements and Links.
What I like best is the way NPM suggests solutions to network problems. Its also has the ability to monitor the health of individual VMware virtual machines. If you are interested in troubleshooting, and creating network maps, then I recommend that you try NPM now.
Invoke-WebRequest Example 1: Convert Miles to Feet
The task is to interrogate a website for information. In this example PowerShell needs the .AllElements property and the .InnerHtml value
Clear-Host$Site = «http://www.asknumbers.com/MilesToFeetConversion.aspx»$Request = Invoke-WebRequest -URI $Site$Request.AllElements | Where-Object {$_.InnerHtml -like «*=*»} |Sort-Object { $_.InnerHtml.Length } | Select-Object InnerText -First 1
innerText ——— 1 Mile = 5280 Feet
Note 3: This URI has an .aspx extension.
Note 4: You can discover more properties by applying Get-Member to $Request.AllElements.
Invoke-WebRequest Example 2: Find Lucky Numbers
The task is to interrogate a website for he latest lucky numbers. In this example PowerShell needs the .AllElements property and the .InnerHtml value
Clear-Host$Site = Invoke-WebRequest -URI «http://www.loto49.ro/arhiva-loto49.php»$LuckyNumbers = $Site.AllElements | Where-Object {$_.tagName -eq «TD»} | ` Where-Object { $_.InnerText -match «^\d+$» } | `Select-Object -property InnerText -Last 6$LuckyNumbers
innerText ——— (Results will update each!)
Note 5: To see the webpage itself, you can append.
$IE=New-Object -com InternetExplorer.Application $IE.Navigate2($Site)$IE.Visible=$true$IE.FullScreen=$true
There are so many good gadgets; it’s like having free rein of a sweetshop. Thankfully the utilities are displayed logically: monitoring, network discovery, diagnostic, and Cisco tools. Try the SolarWinds Engineer’s Toolset now!
Research Invoke-WebRequest
Call for PowerShell’s built-in help to learn more about it’s parameters. Be aware that this only works in version 3.0, previously in PowerShell 2.0 you would have to rely on .NET Framework.
Clear-HostGet-Help Invoke-WebRequest -full
Technically, Invoke-WebRequest sends HTTP and HTTPS requests to a web page or web service. It parses the response and returns collections of forms, links, images, and other HTML elements.
Note 6: The -Method parameter is interesting, it takes values of: Default, Delete, Get, Head, Options, Post, Put, and Trace.
Research Other Members of the Invoke Family
The verb ‘Invoke’ is rarely used in general English or in PowerShell, his is how to research other members of this family:
# PowerShell’s Invoke family of cmdlets Get-Command -Verb invoke
Invoke-Command Invoke-Expression Invoke-History Invoke-ItemInvoke-WebRequest Invoke-WmiMethod Invoke-WSManAction
Summary of PowerShell’s Invoke-WebRequest
Invoke-WebRequest is an interesting cmdlet introduced in PowerShell 3.0. I have examples showing how to investigate web pages.
If you like this page then please share it with your friends
See more Microsoft PowerShell tasks:
• PowerShell Home • Shell Application • New-Object • PowerShell Add Printer • PowerShell -com
• PowerShell Logon Script • Map Network Drive • PowerShell Create Shortcut • Free CSV Import Tool
• Invoke-Expression • Invoke-Command • Invoke-Item • PowerShell Expression v Command Mode
Powershell Download Zip File
The method to download zip files is pretty much the same as a normal file. But I wanted to show you how that downloads and extracts the zip file. This way you can immediately process the files inside the zip file without manual interaction.
I am going to use this sample csv on GitHub which we can download in a zip file. We have to set a destination for the zip file itself and a path where we want to extract the files to.
The Invoke-WebRequest downloads the zip file just like any other file.
# URL and Destination $url = "https://github.com/datapackage-examples/sample-csv/archive/refs/heads/master.zip" # Create temp destination for the zip and get zipfile name from source URL $zipFile = "c:\temp\zipfiles" + $(Split-Path -Path $Url -Leaf) # Extract path $extractPath = "c:\temp\zipfiles" # Download file Invoke-WebRequest -Uri $url -OutFile $zipFile
The next step is to extract the zip file automatically in the desired location. For this we are going to use a COM object. With the COM object we can extract the zip file and copy the content to the desired location.
# Create instance of COM Object $objShell = New-Object -ComObject Shell.Application # Extract the Files $extractedFiles = $ObjShell.NameSpace($zipFile).Items() # Copy the extracted files to the destination folder $ObjShell.NameSpace($extractPath).CopyHere($extractedFiles)