Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
Question
Friday, December 30, 2016 9:21 AM
Hi guys,
I am trying to download a file from a webpage. The webpage redirects the link to several links. I have tried using Webclient as well as httpwebrequest. I am able to create an 'instance' of the downloaded file in the given location locally but it doesn't contain anything but the text "Temporary Redirect". I have also tried using cookiecontainer in both webclient as well as httpwebrequest. Also I have tried My.Computer.Network.DownloadFile(<address>, file path), but with the same result.
This is my webclient code
Dim ss As String
ss = "http://somewebsitewithawordfilelink.doc"
Dim client As New WebClient
client = New WebClient
client.UseDefaultCredentials = True
client.Headers.Add(HttpRequestHeader.Cookie, "cookievalue")
client.DownloadFile(ss, txtFolder.Text & "\gg.doc")
This is my httpwebrequest code
Dim ss as string
ss= "http://somewebsitewithawordfilelink"
Dim httpRequest As HttpWebRequest = DirectCast(WebRequest.Create(ss), HttpWebRequest)
httpRequest.Method = WebRequestMethods.Http.[Get]
httpRequest.UseDefaultCredentials = True
httpRequest.AllowAutoRedirect = False
Dim cookieContainer As CookieContainer = New CookieContainer()
httpRequest.CookieContainer = cookieContainer
Dim httpResponse As HttpWebResponse = DirectCast(httpRequest.GetResponse(), HttpWebResponse)
Dim httpResponseStream As Stream = httpResponse.GetResponseStream()
Using fs = File.Create("C:\Users\xxxx\Desktop\Dump\tt.doc")
httpResponseStream.CopyTo(fs)
End Using
Any ideas on this are appreciated. Both my attempts end up in a file containing the text "Temporary Redirect".
Thanks in advance.
Kish
Kish Learning never ends!! Please don't forget to click "Mark As Answer" for the post in which you found your solution.
All replies (29)
Wednesday, January 4, 2017 2:31 PM âś…Answered
Frank,
I put a breakpoint in that sub. I observed that one of the redirected links in the challengedURI was directed towards a link containing javascript. Is it possible that the redirection is being handled by js at some point which doesn't allow the webrequest to reach the desired URL?
//Kish
Kish Learning never ends!! Please don't forget to click "Mark As Answer" for the post in which you found your solution.
As previously mentioned "And doing anything other than working with the URL you are trying to use would probably be a waste of time.". And since a WebBrowser works then perhaps try using a WebBrowser control to see if that works. If WebBrowser control works then dynamically create WebBrowser control and use code to catch download prompt and redirect to your own code so bypass download prompt.
But without the URL you use it is not possible to assist you really.
La vida loca
Friday, December 30, 2016 9:29 AM
Did you first try to download the file without a program.
If you cannot do it by hand, you cannot do it with a program.
Success
Cor
Friday, December 30, 2016 9:31 AM
Did you first try to download the file without a program.
If you cannot do it by hand, you cannot do it with a program.
Success
Cor
Thanks for replying. Yes I did and I am able to download the file.
Kish Learning never ends!! Please don't forget to click "Mark As Answer" for the post in which you found your solution.
Friday, December 30, 2016 9:44 AM
Then this should be all you need to download the file
Call (New WebClient).DownloadFile(The full Url as string, The full File Path where to save)
Use that to download the file without all nonsense as cookie containers and not real fixed names.
If that runs, you can make it as you wish
Success
Cor
Friday, December 30, 2016 9:49 AM
Then this should be all you need to download the file
Call (New WebClient).DownloadFile(The full Url as string, The full File Path where to save)Use that to download the file without all nonsense as cookie containers and not real fixed names.
If that runs, you can make it as you wish
Success
Cor
Did that. After a few seconds I get an exception "Too many automatic redirections were attempted".
Kish Learning never ends!! Please don't forget to click "Mark As Answer" for the post in which you found your solution.
Friday, December 30, 2016 10:45 AM
Then you must get the same response if you download it by hand.
Success
Cor
Friday, December 30, 2016 11:44 AM
Then you must get the same response if you download it by hand.
Success
Cor
It asks me to save the file and it saves the original file with all the contents properly. While using the method you mentioned it gives me a file with the text "Temporary Redirect" written and nothing else with file size 2kb.
Kish Learning never ends!! Please don't forget to click "Mark As Answer" for the post in which you found your solution.
Friday, December 30, 2016 11:49 AM
Then you must get the same response if you download it by hand.
Success
CorIt asks me to save the file and it saves the original file with all the contents properly. While using the method you mentioned it gives me a file with the text "Temporary Redirect" written and nothing else with file size 2kb.
Kish Learning never ends!! Please don't forget to click "Mark As Answer" for the post in which you found your solution.
and you are sure the by hand downloaded file is not from a redirected page?
Success
Cor
Friday, December 30, 2016 12:04 PM
Then you must get the same response if you download it by hand.
Success
CorIt asks me to save the file and it saves the original file with all the contents properly. While using the method you mentioned it gives me a file with the text "Temporary Redirect" written and nothing else with file size 2kb.
Kish Learning never ends!! Please don't forget to click "Mark As Answer" for the post in which you found your solution.
and you are sure the by hand downloaded file is not from a redirected page?
Success
Cor
Yes I am sure. On the same link when I click by hand it downloads the file to the location I save whereas when I use the same link for the method you mentioned I get the error "Too many automatic redirections were attempted" and the procedure fails there.
Kish Learning never ends!! Please don't forget to click "Mark As Answer" for the post in which you found your solution.
Friday, December 30, 2016 12:17 PM
Yes I am sure. On the same link when I click by hand it downloads the file to the location I save
Not place where you download too, that is obvious. The place where you get it from
Success
Cor
Friday, December 30, 2016 12:21 PM
Yes I am sure. On the same link when I click by hand it downloads the file to the location I save
Not place where you download too, that is obvious. The place where you get it from
Success
Cor
I am not sure what you are asking there. I use the same link to download the file using the browser and the same link I am using in the code.
Kish Learning never ends!! Please don't forget to click "Mark As Answer" for the post in which you found your solution.
Friday, December 30, 2016 2:31 PM
Hi
Change your code to allow for redirection and set a max-redirection count
httpWebRequest.MaximumAutomaticRedirections=100;
httpWebRequest.AllowAutoRedirect=true;
Regards,
Koen
Monday, January 2, 2017 4:16 AM
Still the sam result. To download the file I am going with My.Computer.Network.DownloadFile("<address>", local path). Is it the correct way to doanload the file using httpwebrequest?
Kish Learning never ends!! Please don't forget to click "Mark As Answer" for the post in which you found your solution.
Monday, January 2, 2017 6:48 AM
Yes I am sure. On the same link when I click by hand it downloads the file to the location I save
Not place where you download too, that is obvious. The place where you get it from
Success
CorI am not sure what you are asking there. I use the same link to download the file using the browser and the same link I am using in the code.
Kish Learning never ends!! Please don't forget to click "Mark As Answer" for the post in which you found your solution.
Obviously the browser isn't written with the code you are attempting to use since the browser works and your code does not.
If you search the net for HTTPWebRequest Redirect you will find numerous threads about it and how to deal with it.
Since I have no idea what url you try to use to download some file, and if it requires an account it wouldn't matter anyhow, then there really isn't much way to test code to see if it will work with the url you try to download some file via.
La vida loca
Monday, January 2, 2017 6:53 AM
Yes I am sure. On the same link when I click by hand it downloads the file to the location I save
Not place where you download too, that is obvious. The place where you get it from
Success
CorI am not sure what you are asking there. I use the same link to download the file using the browser and the same link I am using in the code.
Kish Learning never ends!! Please don't forget to click "Mark As Answer" for the post in which you found your solution.
Obviously the browser isn't written with the code you are attempting to use since the browser works and your code does not.
If you search the net for HTTPWebRequest Redirect you will find numerous threads about it and how to deal with it.
Since I have no idea what url you try to use to download some file, and if it requires an account it wouldn't matter anyhow, then there really isn't much way to test code to see if it will work with the url you try to download some file via.
La vida loca
I agree with your point. I have encountered this error since there was a change in the webserver from where I am trying to download the files. Earlier they used to be static links which worked fine, now they have been replaced with a redirection, hence this error.
One interesting thing I observed today, If I copy paste the link in the browser it gives a 404 Error, however when I paste it again, it starts downloading the file. Should I introduce a delay?
Kish Learning never ends!! Please don't forget to click "Mark As Answer" for the post in which you found your solution.
Monday, January 2, 2017 2:46 PM
Yes I am sure. On the same link when I click by hand it downloads the file to the location I save
Not place where you download too, that is obvious. The place where you get it from
Success
CorI am not sure what you are asking there. I use the same link to download the file using the browser and the same link I am using in the code.
Kish Learning never ends!! Please don't forget to click "Mark As Answer" for the post in which you found your solution.
Obviously the browser isn't written with the code you are attempting to use since the browser works and your code does not.
If you search the net for HTTPWebRequest Redirect you will find numerous threads about it and how to deal with it.
Since I have no idea what url you try to use to download some file, and if it requires an account it wouldn't matter anyhow, then there really isn't much way to test code to see if it will work with the url you try to download some file via.
La vida loca
I agree with your point. I have encountered this error since there was a change in the webserver from where I am trying to download the files. Earlier they used to be static links which worked fine, now they have been replaced with a redirection, hence this error.
One interesting thing I observed today, If I copy paste the link in the browser it gives a 404 Error, however when I paste it again, it starts downloading the file. Should I introduce a delay?
Kish Learning never ends!! Please don't forget to click "Mark As Answer" for the post in which you found your solution.
You're grasping at straws. A WebBrowser is written by a programmer. A WebBrowser is written to deal with redirects. Your code is not a WebBrowsers code and apparently isn't written properly to work with redirects. Once you have that accomplished such that you eventually retrieve the URL that will be the final URL for downloading a file then you can worry about how to download the file.
As I previously mentioned you should search the net for HTTPWebRequest redirect to see how others handle redirects with HTTPWebRequest. Most search results I've seen were in C#. But reading them provides ideas on how to accomplish the task.
Some search results even mention to use fiddler so you can monitor the packets between your system and the site you are using HTTPWebRequest with and a browser with and then read the packet headers to see what is occuring since the browser works and HTTPWebRequest does not. You can also use WireShark for that.
And since you are not providing a link to the file you want to download nobody can really assist you with this anyhow as all redirects may not be the same for all I know. And doing anything other than working with the URL you are trying to use would probably be a waste of time.
La vida loca
Monday, January 2, 2017 3:26 PM
Kishlaya,
Try this but do note this is just a test really.
It's set up to get text (as from a text file). Put the URL in the appropriate place shown below, run it, and let's see what comes back (if anything):
Option Strict On
Option Explicit On
Option Infer Off
Imports System.IO
Imports System.Net
Public Class Form1
Private Sub Form1_Load(ByVal sender As System.Object, _
ByVal e As System.EventArgs) _
Handles MyBase.Load
Const url As String = _
"Put The Actual URL Here"
Dim result As String = GetTextFromURL(url)
If Not String.IsNullOrWhiteSpace(result) Then
Stop
End If
End Sub
Private Function GetTextFromURL(ByVal url As String) As String
Dim retVal As String = ""
Try
Dim request As WebRequest = WebRequest.Create(url)
Using response As HttpWebResponse = DirectCast(request.GetResponse, HttpWebResponse)
Using dataStream As Stream = response.GetResponseStream
Using reader As New StreamReader(dataStream)
Dim responseFromServer As String = reader.ReadToEnd()
retVal = responseFromServer
End Using
End Using
End Using
Catch ex As Exception
retVal = ""
End Try
Return retVal
End Function
End Class
"One who has no vices also has no virtues..."
Monday, January 2, 2017 4:13 PM
Kishlaya,
You might also want to try this:
Option Strict On
Option Explicit On
Option Infer Off
Imports System.IO
Imports System.IO.Path
Imports System.Net
Imports System.Environment
Public Class Form1
Private Sub Form1_Load(ByVal sender As System.Object, _
ByVal e As System.EventArgs) _
Handles MyBase.Load
Const url As String = _
"http://www.fls-online.com/VBNet_Forum/01-02-17/ScreenShot01.png"
Dim desktop As String = _
GetFolderPath(Environment.SpecialFolder.Desktop)
Dim outputPath As String = _
Combine(desktop, "TestMe.bin")
WriteFileFromURL(url, outputPath)
Stop
End Sub
Private Function GetTextFromURL(ByVal url As String) As String
Dim retVal As String = ""
Try
Dim request As WebRequest = WebRequest.Create(url)
Using response As HttpWebResponse = DirectCast(request.GetResponse, HttpWebResponse)
Using dataStream As Stream = response.GetResponseStream
Using reader As New StreamReader(dataStream)
Dim responseFromServer As String = reader.ReadToEnd()
retVal = responseFromServer
End Using
End Using
End Using
Catch ex As Exception
retVal = ""
End Try
Return retVal
End Function
Private Sub _
WriteFileFromURL(ByVal url As String, _
ByVal filePath As String)
Dim request As WebRequest = WebRequest.Create(url)
request.Timeout = 1500
Using response As HttpWebResponse = DirectCast(request.GetResponse, HttpWebResponse)
Using dataStream As Stream = response.GetResponseStream
If dataStream IsNot Nothing Then
Using fs As New FileStream(filePath, FileMode.Create)
dataStream.CopyTo(fs)
End Using
End If
End Using
End Using
End Sub
End Class
It should create new file and put it on your desktop:
If you'll then open that file in a text editor (like Notepad), you'll see the data:
If all of that works, delete the file from your desktop, modify the URL to your actual one and try it again.
If it works with your actual URL then I'll set something up to work with it after that. Obviously it only knows that it's "data" and my new method just writes the stream to a file, so we'll have to modify it once its that far.
Let me know please?
"One who has no vices also has no virtues..."
Tuesday, January 3, 2017 6:11 AM
Kishlaya,
You might also want to try this:
Option Strict On Option Explicit On Option Infer Off Imports System.IO Imports System.IO.Path Imports System.Net Imports System.Environment Public Class Form1 Private Sub Form1_Load(ByVal sender As System.Object, _ ByVal e As System.EventArgs) _ Handles MyBase.Load Const url As String = _ "http://www.fls-online.com/VBNet_Forum/01-02-17/ScreenShot01.png" Dim desktop As String = _ GetFolderPath(Environment.SpecialFolder.Desktop) Dim outputPath As String = _ Combine(desktop, "TestMe.bin") WriteFileFromURL(url, outputPath) Stop End Sub Private Function GetTextFromURL(ByVal url As String) As String Dim retVal As String = "" Try Dim request As WebRequest = WebRequest.Create(url) Using response As HttpWebResponse = DirectCast(request.GetResponse, HttpWebResponse) Using dataStream As Stream = response.GetResponseStream Using reader As New StreamReader(dataStream) Dim responseFromServer As String = reader.ReadToEnd() retVal = responseFromServer End Using End Using End Using Catch ex As Exception retVal = "" End Try Return retVal End Function Private Sub _ WriteFileFromURL(ByVal url As String, _ ByVal filePath As String) Dim request As WebRequest = WebRequest.Create(url) request.Timeout = 1500 Using response As HttpWebResponse = DirectCast(request.GetResponse, HttpWebResponse) Using dataStream As Stream = response.GetResponseStream If dataStream IsNot Nothing Then Using fs As New FileStream(filePath, FileMode.Create) dataStream.CopyTo(fs) End Using End If End Using End Using End Sub End ClassIt should create new file and put it on your desktop:
If you'll then open that file in a text editor (like Notepad), you'll see the data:
If all of that works, delete the file from your desktop, modify the URL to your actual one and try it again.
If it works with your actual URL then I'll set something up to work with it after that. Obviously it only knows that it's "data" and my new method just writes the stream to a file, so we'll have to modify it once its that far.
Let me know please?
"One who has no vices also has no virtues..."
Hello Frank,
Thank you for all that effort. It worked as you described for the link you provided for the png.
When I used it with my actual URL, I am getting the error "The operation has timed out" In the WriteFileFromURL Sub.
Kish Learning never ends!! Please don't forget to click "Mark As Answer" for the post in which you found your solution.
Tuesday, January 3, 2017 11:33 AM
When I used it with my actual URL, I am getting the error "The operation has timed out" In the WriteFileFromURL Sub.
Kish Learning never ends!! Please don't forget to click "Mark As Answer" for the post in which you found your solution.
Kishlaya,
How about put a breakpoint in that sub, step into each line, and see just what it's doing.
If it really is just a time-out issue, then modify it a bit: In the sub you'll see that it creates a WebRequest. Change that to this please:
Dim request As HttpWebRequest = _
DirectCast(HttpWebRequest.Create(url), HttpWebRequest)
With request
.Timeout = 5000
.MaximumAutomaticRedirections = 1000
.AllowAutoRedirect = True
End With
Then try it and see if it works. Put the breakpoint in first though - that's what will tell you what's going on.
"One who has no vices also has no virtues..."
Wednesday, January 4, 2017 4:19 AM
Frank,
I put a breakpoint in that sub. I observed that one of the redirected links in the challengedURI was directed towards a link containing javascript. Is it possible that the redirection is being handled by js at some point which doesn't allow the webrequest to reach the desired URL?
//Kish
Kish Learning never ends!! Please don't forget to click "Mark As Answer" for the post in which you found your solution.
Wednesday, January 4, 2017 11:11 AM
Is it possible that the redirection is being handled by js at some point which doesn't allow the webrequest to reach the desired URL?
I suppose that's possible but I'm out of my bailiwick to know, honestly.
In the first test that I posted the other day (the one that just reads it as text), did it succeed or did nothing come through?
"One who has no vices also has no virtues..."
Thursday, January 5, 2017 3:16 AM
Frank,
I put a breakpoint in that sub. I observed that one of the redirected links in the challengedURI was directed towards a link containing javascript. Is it possible that the redirection is being handled by js at some point which doesn't allow the webrequest to reach the desired URL?
//Kish
Kish Learning never ends!! Please don't forget to click "Mark As Answer" for the post in which you found your solution.
As previously mentioned "And doing anything other than working with the URL you are trying to use would probably be a waste of time.". And since a WebBrowser works then perhaps try using a WebBrowser control to see if that works. If WebBrowser control works then dynamically create WebBrowser control and use code to catch download prompt and redirect to your own code so bypass download prompt.
But without the URL you use it is not possible to assist you really.
La vida loca
I understand that without the URL it is not possible to help, moreover since the URL is an internal (company) one it will not be of much help. You are probably right I hope. I'll try with a web browser control
Thanks for the help so far.
//Kish
Kish Learning never ends!! Please don't forget to click "Mark As Answer" for the post in which you found your solution.
Thursday, January 5, 2017 3:17 AM
Is it possible that the redirection is being handled by js at some point which doesn't allow the webrequest to reach the desired URL?
I suppose that's possible but I'm out of my bailiwick to know, honestly.
In the first test that I posted the other day (the one that just reads it as text), did it succeed or did nothing come through?
"One who has no vices also has no virtues..."
The first test you posted came through and the file was downloaded as desired (TestMe.bin). Thanks for the help so far.
//Kish
Kish Learning never ends!! Please don't forget to click "Mark As Answer" for the post in which you found your solution.
Thursday, January 5, 2017 11:07 AM
The first test you posted came through and the file was downloaded as desired (TestMe.bin). Thanks for the help so far.
//Kish
Kish Learning never ends!! Please don't forget to click "Mark As Answer" for the post in which you found your solution.
If you'll scroll up some you'll see the first post from me that has a function named "GetTextFromURL".
I'm wondering if you tried that and if so what (if anything) was returned?
"One who has no vices also has no virtues..."
Friday, January 6, 2017 10:03 AM
The first test you posted came through and the file was downloaded as desired (TestMe.bin). Thanks for the help so far.
//Kish
Kish Learning never ends!! Please don't forget to click "Mark As Answer" for the post in which you found your solution.
If you'll scroll up some you'll see the first post from me that has a function named "GetTextFromURL".
I'm wondering if you tried that and if so what (if anything) was returned?
"One who has no vices also has no virtues..."
It was stuck at time out error on
Using response As HttpWebResponse = DirectCast(request.GetResponse, HttpWebResponse)
Kish Learning never ends!! Please don't forget to click "Mark As Answer" for the post in which you found your solution.
Saturday, January 7, 2017 9:04 AM
It was stuck at time out error on
Using response As HttpWebResponse = DirectCast(request.GetResponse, HttpWebResponse)Kish Learning never ends!! Please don't forget to click "Mark As Answer" for the post in which you found your solution.
Sorry for the delay...
*****
This is what comes to mind here: Find out if they (whatever company is hosting the file(s) that you're trying to download) have an API.
This is reminiscent of how some of them (Google is one of them, but not the only one) set their services up: They not only make it mandatory that you use their API and no other way, they periodically change things around so that even if you do figure out some other way in, it won't work in a few days or weeks because they've changed it.
"One who has no vices also has no virtues..."
Tuesday, January 10, 2017 11:24 AM
It was stuck at time out error on
Using response As HttpWebResponse = DirectCast(request.GetResponse, HttpWebResponse)Kish Learning never ends!! Please don't forget to click "Mark As Answer" for the post in which you found your solution.
Sorry for the delay...
*****
This is what comes to mind here: Find out if they (whatever company is hosting the file(s) that you're trying to download) have an API.
This is reminiscent of how some of them (Google is one of them, but not the only one) set their services up: They not only make it mandatory that you use their API and no other way, they periodically change things around so that even if you do figure out some other way in, it won't work in a few days or weeks because they've changed it.
"One who has no vices also has no virtues..."
I am trying to get all the information from them. Hopfully I will get some solution. So far the web browser control looks closer to the solution to the problem (although not complete).
Kish Learning never ends!! Please don't forget to click "Mark As Answer" for the post in which you found your solution.
Wednesday, January 11, 2017 6:39 AM
It was stuck at time out error on
Using response As HttpWebResponse = DirectCast(request.GetResponse, HttpWebResponse)Kish Learning never ends!! Please don't forget to click "Mark As Answer" for the post in which you found your solution.
Sorry for the delay...
*****
This is what comes to mind here: Find out if they (whatever company is hosting the file(s) that you're trying to download) have an API.
This is reminiscent of how some of them (Google is one of them, but not the only one) set their services up: They not only make it mandatory that you use their API and no other way, they periodically change things around so that even if you do figure out some other way in, it won't work in a few days or weeks because they've changed it.
"One who has no vices also has no virtues..."
I am trying to get all the information from them. Hopfully I will get some solution. So far the web browser control looks closer to the solution to the problem (although not complete).
Kish Learning never ends!! Please don't forget to click "Mark As Answer" for the post in which you found your solution.
Well I have code for the WebBrowser control which is a class or possibly two which when a WebBrowser control goes to download a file this code will be set as the download capability so there is no dialog displayed by the WebBrowser control. Plus this code gets the URL of the file to be downloaded just as the dialog does.
So if the WebBrowser control is working for you and it redirects to the final location to download a file and then the download dialog appears requiring manual intervention to select save or save as then the code I have will probably work fine.
And then you can probably instantiate a WebBrowser control programmatically, never display it, have it navigate to download the file and instead of a dialog appearing programmatically save the file somewhere.
I got the code from here Implement IDownloadManger for my application only a while back and implemented it in VB and I have it so if the WebBrowser navigates to a file download then a SaveFileDialog pops up and a WebClient downloads the file saving it to the path in the SaveFileDialog. So I can provide the code if you want it. Probably a project would be best so you could see how it works.
So if a WebBrowser control can follow all the re-directs to the final file download link and at that point a dialog displays to save/save as or cancel the download then the code I have could be altered to auto-save a file which would be auto-downloaded from the final download link but would have to be tested to verify it would work.
As you can see in the image, although you may need to zoom in with your browser, the Form has the URL "https://www.microsoft.com/en-us/download/confirmation.aspx?id=15039" which is the download link for the link displayed in the SaveFileDialogs title so I suppose it is a redirect URL to the URL displayed in the title of the SaveFileDialog. Since the code for the SaveFileDialog isn't necessary code to auto download a file, using the file name used in the final URL, to save the file to some known location on the system using a WebClient to download the file is available.
La vida loca