pedrosaclout
/
instance_scrape
0
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

working almost perfect, still breaks on anchors sometimes

Browse Source
master
Pedro Sá Couto 5 years ago
parent e52ec2dc37
commit 1e8e862ea6
2 changed files with 200 additions and 15 deletions
Show Stats Download Patch File Download Diff File

189
geckodriver.log
Unescape Escape View File

@ -2651,3 +2651,192 @@ Exiting due to channel error.
Exiting due to channel error.
Exiting due to channel error.
Exiting due to channel error.
1552752823314 mozrunner::runner INFO Running command: "/Applications/Firefox.app/Contents/MacOS/firefox-bin" "-marionette" "-foreground" "-no-remote" "-profile" "/var/folders/bc/txjl5mmd7cb1ngc8vvs687sc0000gn/T/rust_mozprofile.3DKdFIMfEOJM"
1552752823730 addons.webextension.screenshots@mozilla.org WARN Loading extension 'screenshots@mozilla.org': Reading manifest: Invalid host permission: resource://pdf.js/
1552752823731 addons.webextension.screenshots@mozilla.org WARN Loading extension 'screenshots@mozilla.org': Reading manifest: Invalid host permission: about:reader*
Can't find symbol 'GetGraphicsResetStatus'.
1552752825108 Marionette INFO Listening on port 53822
1552752825170 Marionette WARN TLS certificate errors will be ignored for this session
1552753000086 mozrunner::runner INFO Running command: "/Applications/Firefox.app/Contents/MacOS/firefox-bin" "-marionette" "-foreground" "-no-remote" "-profile" "/var/folders/bc/txjl5mmd7cb1ngc8vvs687sc0000gn/T/rust_mozprofile.FvDFUg9EOY33"
1552753000489 addons.webextension.screenshots@mozilla.org WARN Loading extension 'screenshots@mozilla.org': Reading manifest: Invalid host permission: resource://pdf.js/
1552753000489 addons.webextension.screenshots@mozilla.org WARN Loading extension 'screenshots@mozilla.org': Reading manifest: Invalid host permission: about:reader*
Can't find symbol 'GetGraphicsResetStatus'.
1552753001869 Marionette INFO Listening on port 53925
1552753001927 Marionette WARN TLS certificate errors will be ignored for this session
[GFX1-]: Receive IPC close with reason=AbnormalShutdown
Exiting due to channel error.
Exiting due to channel error.
Exiting due to channel error.
[GFX1-]: Receive IPC close with reason=AbnormalShutdown
Exiting due to channel error.
1552753049215 mozrunner::runner INFO Running command: "/Applications/Firefox.app/Contents/MacOS/firefox-bin" "-marionette" "-foreground" "-no-remote" "-profile" "/var/folders/bc/txjl5mmd7cb1ngc8vvs687sc0000gn/T/rust_mozprofile.rpJ1QBAj04oZ"
1552753049609 addons.webextension.screenshots@mozilla.org WARN Loading extension 'screenshots@mozilla.org': Reading manifest: Invalid host permission: resource://pdf.js/
1552753049609 addons.webextension.screenshots@mozilla.org WARN Loading extension 'screenshots@mozilla.org': Reading manifest: Invalid host permission: about:reader*
Can't find symbol 'GetGraphicsResetStatus'.
1552753050997 Marionette INFO Listening on port 53983
1552753051059 Marionette WARN TLS certificate errors will be ignored for this session
1552753075848 mozrunner::runner INFO Running command: "/Applications/Firefox.app/Contents/MacOS/firefox-bin" "-marionette" "-foreground" "-no-remote" "-profile" "/var/folders/bc/txjl5mmd7cb1ngc8vvs687sc0000gn/T/rust_mozprofile.ScqclPmhfJXr"
1552753076252 addons.webextension.screenshots@mozilla.org WARN Loading extension 'screenshots@mozilla.org': Reading manifest: Invalid host permission: resource://pdf.js/
1552753076252 addons.webextension.screenshots@mozilla.org WARN Loading extension 'screenshots@mozilla.org': Reading manifest: Invalid host permission: about:reader*
Can't find symbol 'GetGraphicsResetStatus'.
1552753077641 Marionette INFO Listening on port 54037
1552753077681 Marionette WARN TLS certificate errors will be ignored for this session
[GFX1-]: Receive IPC close with reason=AbnormalShutdown
Exiting due to channel error.
[GFX1-]: Receive IPC close with reason=AbnormalShutdown
Exiting due to channel error.
[GFX1-]: Receive IPC close with reason=AbnormalShutdown
Exiting due to channel error.
[GFX1-]: Receive IPC close with reason=AbnormalShutdown
[Child 5688, Chrome_ChildThread] WARNING: pipe error: Broken pipe: file /builds/worker/workspace/build/src/ipc/chromium/src/chrome/common/ipc_channel_posix.cc, line 718
Exiting due to channel error.
1552753148688 mozrunner::runner INFO Running command: "/Applications/Firefox.app/Contents/MacOS/firefox-bin" "-marionette" "-foreground" "-no-remote" "-profile" "/var/folders/bc/txjl5mmd7cb1ngc8vvs687sc0000gn/T/rust_mozprofile.Ic9Zdp4579La"
1552753149092 addons.webextension.screenshots@mozilla.org WARN Loading extension 'screenshots@mozilla.org': Reading manifest: Invalid host permission: resource://pdf.js/
1552753149092 addons.webextension.screenshots@mozilla.org WARN Loading extension 'screenshots@mozilla.org': Reading manifest: Invalid host permission: about:reader*
Can't find symbol 'GetGraphicsResetStatus'.
1552753150486 Marionette INFO Listening on port 54098
1552753150528 Marionette WARN TLS certificate errors will be ignored for this session
console.error: BroadcastService:
receivedBroadcastMessage: handler for
remote-settings/monitor_changes
threw error:
Message: Error: Polling for changes failed: NetworkError when attempting to fetch resource..
Stack:
remoteSettingsFunction/remoteSettings.pollChanges@resource://services-settings/remote-settings.js:188:13
console.error: BroadcastService:
receivedBroadcastMessage: handler for
remote-settings/monitor_changes
threw error:
Message: Error: Polling for changes failed: NetworkError when attempting to fetch resource..
Stack:
remoteSettingsFunction/remoteSettings.pollChanges@resource://services-settings/remote-settings.js:188:13
console.error: BroadcastService:
receivedBroadcastMessage: handler for
remote-settings/monitor_changes
threw error:
Message: Error: Polling for changes failed: NetworkError when attempting to fetch resource..
Stack:
remoteSettingsFunction/remoteSettings.pollChanges@resource://services-settings/remote-settings.js:188:13
JavaScript error: https://mstdn.komittee.net/packs/js/common-0c5fef3f4c060b317d69.js, line 1: TypeError: t is undefined
JavaScript error: https://mstdn.komittee.net/packs/js/common-0c5fef3f4c060b317d69.js, line 1: TypeError: this is undefined
[GFX1-]: Receive IPC close with reason=AbnormalShutdown
[GFX1-]: Receive IPC close with reason=AbnormalShutdown
Exiting due to channel error.
Exiting due to channel error.
[GFX1-]: Receive IPC close with reason=AbnormalShutdown
[Child 5705, Chrome_ChildThread] WARNING: pipe error: Broken pipe: file /builds/worker/workspace/build/src/ipc/chromium/src/chrome/common/ipc_channel_posix.cc, line 718
Exiting due to channel error.
1552753254336 mozrunner::runner INFO Running command: "/Applications/Firefox.app/Contents/MacOS/firefox-bin" "-marionette" "-foreground" "-no-remote" "-profile" "/var/folders/bc/txjl5mmd7cb1ngc8vvs687sc0000gn/T/rust_mozprofile.G10oVxbCNtEX"
1552753254733 addons.webextension.screenshots@mozilla.org WARN Loading extension 'screenshots@mozilla.org': Reading manifest: Invalid host permission: resource://pdf.js/
1552753254733 addons.webextension.screenshots@mozilla.org WARN Loading extension 'screenshots@mozilla.org': Reading manifest: Invalid host permission: about:reader*
Can't find symbol 'GetGraphicsResetStatus'.
1552753256122 Marionette INFO Listening on port 54176
1552753256180 Marionette WARN TLS certificate errors will be ignored for this session
console.error: BroadcastService:
receivedBroadcastMessage: handler for
remote-settings/monitor_changes
threw error:
Message: Error: Polling for changes failed: NetworkError when attempting to fetch resource..
Stack:
remoteSettingsFunction/remoteSettings.pollChanges@resource://services-settings/remote-settings.js:188:13
JavaScript error: https://mstdn.komittee.net/packs/js/common-0c5fef3f4c060b317d69.js, line 1: TypeError: t is undefined
JavaScript error: https://mstdn.komittee.net/packs/js/common-0c5fef3f4c060b317d69.js, line 1: TypeError: this is undefined
[GFX1-]: Receive IPC close with reason=AbnormalShutdown
Exiting due to channel error.
[GFX1-]: Receive IPC close with reason=AbnormalShutdown
[Child 5735, Chrome_ChildThread] WARNING: pipe error: Broken pipe: file /builds/worker/workspace/build/src/ipc/chromium/src/chrome/common/ipc_channel_posix.cc, line 718
Exiting due to channel error.
[GFX1-]: Receive IPC close with reason=AbnormalShutdown
Exiting due to channel error.
[GFX1-]: Receive IPC close with reason=AbnormalShutdown
Exiting due to channel error.
1552753365874 mozrunner::runner INFO Running command: "/Applications/Firefox.app/Contents/MacOS/firefox-bin" "-marionette" "-foreground" "-no-remote" "-profile" "/var/folders/bc/txjl5mmd7cb1ngc8vvs687sc0000gn/T/rust_mozprofile.WFmSma9ncXDt"
1552753366277 addons.webextension.screenshots@mozilla.org WARN Loading extension 'screenshots@mozilla.org': Reading manifest: Invalid host permission: resource://pdf.js/
1552753366277 addons.webextension.screenshots@mozilla.org WARN Loading extension 'screenshots@mozilla.org': Reading manifest: Invalid host permission: about:reader*
Can't find symbol 'GetGraphicsResetStatus'.
1552753367658 Marionette INFO Listening on port 54249
1552753367712 Marionette WARN TLS certificate errors will be ignored for this session
console.error: BroadcastService:
receivedBroadcastMessage: handler for
remote-settings/monitor_changes
threw error:
Message: Error: Polling for changes failed: NetworkError when attempting to fetch resource..
Stack:
remoteSettingsFunction/remoteSettings.pollChanges@resource://services-settings/remote-settings.js:188:13
JavaScript error: https://mstdn.komittee.net/packs/js/common-0c5fef3f4c060b317d69.js, line 1: TypeError: t is undefined
JavaScript error: https://mstdn.komittee.net/packs/js/common-0c5fef3f4c060b317d69.js, line 1: TypeError: this is undefined
JavaScript error: https://instances.social/list#lang=en&allowed=nudity_nocw,nudity_all,spam&prohibited=&users=, line 791: TypeError: getLanguage(...) is null
1552753492306 Marionette INFO Stopped listening on port 54249
1552753493913 Marionette INFO Stopped listening on port 53983
1552753495254 Marionette INFO Stopped listening on port 53822
1552753593320 mozrunner::runner INFO Running command: "/Applications/Firefox.app/Contents/MacOS/firefox-bin" "-marionette" "-foreground" "-no-remote" "-profile" "/var/folders/bc/txjl5mmd7cb1ngc8vvs687sc0000gn/T/rust_mozprofile.tT07r7fom2UZ"
1552753593736 addons.webextension.screenshots@mozilla.org WARN Loading extension 'screenshots@mozilla.org': Reading manifest: Invalid host permission: resource://pdf.js/
1552753593736 addons.webextension.screenshots@mozilla.org WARN Loading extension 'screenshots@mozilla.org': Reading manifest: Invalid host permission: about:reader*
Can't find symbol 'GetGraphicsResetStatus'.
1552753595123 Marionette INFO Listening on port 54496
1552753595151 Marionette WARN TLS certificate errors will be ignored for this session
[GFX1-]: Receive IPC close with reason=AbnormalShutdown
Exiting due to channel error.
Exiting due to channel error.
[GFX1-]: Receive IPC close with reason=AbnormalShutdown
Exiting due to channel error.
1552753617373 mozrunner::runner INFO Running command: "/Applications/Firefox.app/Contents/MacOS/firefox-bin" "-marionette" "-foreground" "-no-remote" "-profile" "/var/folders/bc/txjl5mmd7cb1ngc8vvs687sc0000gn/T/rust_mozprofile.WBlewuR3zJKF"
1552753617776 addons.webextension.screenshots@mozilla.org WARN Loading extension 'screenshots@mozilla.org': Reading manifest: Invalid host permission: resource://pdf.js/
1552753617776 addons.webextension.screenshots@mozilla.org WARN Loading extension 'screenshots@mozilla.org': Reading manifest: Invalid host permission: about:reader*
Can't find symbol 'GetGraphicsResetStatus'.
1552753619158 Marionette INFO Listening on port 54561
1552753619215 Marionette WARN TLS certificate errors will be ignored for this session
[GFX1-]: Receive IPC close with reason=AbnormalShutdown
Exiting due to channel error.
[GFX1-]: Receive IPC close with reason=AbnormalShutdown
Exiting due to channel error.
[GFX1-]: Receive IPC close with reason=AbnormalShutdown
Exiting due to channel error.
1552753727419 mozrunner::runner INFO Running command: "/Applications/Firefox.app/Contents/MacOS/firefox-bin" "-marionette" "-foreground" "-no-remote" "-profile" "/var/folders/bc/txjl5mmd7cb1ngc8vvs687sc0000gn/T/rust_mozprofile.oG6tQi1KMh8p"
1552753727811 addons.webextension.screenshots@mozilla.org WARN Loading extension 'screenshots@mozilla.org': Reading manifest: Invalid host permission: resource://pdf.js/
1552753727811 addons.webextension.screenshots@mozilla.org WARN Loading extension 'screenshots@mozilla.org': Reading manifest: Invalid host permission: about:reader*
Can't find symbol 'GetGraphicsResetStatus'.
1552753729204 Marionette INFO Listening on port 54742
1552753729267 Marionette WARN TLS certificate errors will be ignored for this session
[GFX1-]: Receive IPC close with reason=AbnormalShutdown
Exiting due to channel error.
[GFX1-]: Receive IPC close with reason=AbnormalShutdown
[GFX1-]: Receive IPC close with reason=AbnormalShutdown
Exiting due to channel error.
Exiting due to channel error.
1552753748629 mozrunner::runner INFO Running command: "/Applications/Firefox.app/Contents/MacOS/firefox-bin" "-marionette" "-foreground" "-no-remote" "-profile" "/var/folders/bc/txjl5mmd7cb1ngc8vvs687sc0000gn/T/rust_mozprofile.TOhgY64y2G4R"
1552753749028 addons.webextension.screenshots@mozilla.org WARN Loading extension 'screenshots@mozilla.org': Reading manifest: Invalid host permission: resource://pdf.js/
1552753749028 addons.webextension.screenshots@mozilla.org WARN Loading extension 'screenshots@mozilla.org': Reading manifest: Invalid host permission: about:reader*
Can't find symbol 'GetGraphicsResetStatus'.
1552753750423 Marionette INFO Listening on port 54790
1552753750458 Marionette WARN TLS certificate errors will be ignored for this session
[GFX1-]: Receive IPC close with reason=AbnormalShutdown
Exiting due to channel error.
Exiting due to channel error.
[GFX1-]: Receive IPC close with reason=AbnormalShutdown
Exiting due to channel error.
1552753768395 mozrunner::runner INFO Running command: "/Applications/Firefox.app/Contents/MacOS/firefox-bin" "-marionette" "-foreground" "-no-remote" "-profile" "/var/folders/bc/txjl5mmd7cb1ngc8vvs687sc0000gn/T/rust_mozprofile.DMZezbeQNIie"
1552753768792 addons.webextension.screenshots@mozilla.org WARN Loading extension 'screenshots@mozilla.org': Reading manifest: Invalid host permission: resource://pdf.js/
1552753768793 addons.webextension.screenshots@mozilla.org WARN Loading extension 'screenshots@mozilla.org': Reading manifest: Invalid host permission: about:reader*
Can't find symbol 'GetGraphicsResetStatus'.
1552753770189 Marionette INFO Listening on port 54844
1552753770236 Marionette WARN TLS certificate errors will be ignored for this session
JavaScript error: https://instances.social/list#lang=en&allowed=nudity_nocw,nudity_all,spam&prohibited=&users=, line 791: TypeError: getLanguage(...) is null
JavaScript error: https://instances.social/js/bootstrap.min.js, line 6: Error: Modal is transitioning
JavaScript error: https://instances.social/js/bootstrap.min.js, line 6: Error: Modal is transitioning
JavaScript error: https://instances.social/list#lang=en&allowed=nudity_nocw,nudity_all,spam&prohibited=&users=, line 791: TypeError: getLanguage(...) is null
JavaScript error: https://instances.social/js/bootstrap.min.js, line 6: Error: Modal is transitioning
JavaScript error: https://instances.social/js/bootstrap.min.js, line 6: Error: Modal is transitioning
JavaScript error: https://instances.social/js/bootstrap.min.js, line 6: Error: Modal is transitioning
JavaScript error: https://instances.social/js/bootstrap.min.js, line 6: Error: Modal is transitioning
JavaScript error: https://instances.social/js/bootstrap.min.js, line 6: Error: Modal is transitioning
JavaScript error: https://instances.social/js/bootstrap.min.js, line 6: Error: Modal is transitioning
JavaScript error: https://instances.social/js/bootstrap.min.js, line 6: Error: Modal is transitioning
JavaScript error: https://instances.social/js/bootstrap.min.js, line 6: Error: Modal is transitioning
JavaScript error: https://instances.social/js/bootstrap.min.js, line 6: Error: Modal is transitioning
1552753827252 Marionette INFO Stopped listening on port 54844

26
selenium_instances.py
Unescape Escape View File

@ -6,6 +6,7 @@ import time
import datetime
from pprint import pprint
import requests
import multiprocessing
today = datetime.date.today()
text_file = open("results.txt", "a+")
@ -26,7 +27,6 @@ time.sleep(3)
d = 1
i = 0
p = -1
while True:
@ -43,19 +43,16 @@ while True:
print(description.text)
text_file.write("Description: "+"\n"+description.text+"\n"+"\n")
time.sleep(1)
time.sleep(0.5)
# open instance in new tab
driver.find_element_by_css_selector('#modalInstanceInfo-btn-go').send_keys(Keys.COMMAND + Keys.ENTER)
time.sleep(2)
time.sleep(0.5)
#go to new tab
print([p])
driver.switch_to.window(driver.window_handles[p])
p = p-1
driver.switch_to.window(driver.window_handles[-1])
time.sleep(3)
time.sleep(1)
try:
# get the image source
@ -66,17 +63,16 @@ while True:
if Picture_request.status_code == 200:
with open("image%i.jpg"%i, 'wb') as f:
f.write(Picture_request.content)
print("Printed Image")
except:
#back to original tab
driver.switch_to.window(driver.window_handles[0])
print("Impossible to print image")
time.sleep(1)
time.sleep(0.5)
# # NOT WORKING
# # close new tab
# driver.find_element_by_xpath('/html/body').send_keys(Keys.COMMAND + 'w')
# print("Closing Window")
# close new tab
driver.close()
print("Closing Window")
#back to original tab
driver.switch_to.window(driver.window_handles[0])

Write Preview
Loading…
Cancel
Save