pedrosaclout
/
instance_scrape
0
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

working almost perfect, still breaks on anchors sometimes

Browse Source
master
Pedro Sá Couto 5 years ago
parent e52ec2dc37
commit 1e8e862ea6
2 changed files with 200 additions and 15 deletions
Show Stats Download Patch File Download Diff File

189
geckodriver.log
Unescape Escape View File

@ -2651,3 +2651,192 @@ Exiting due to channel error.
Exiting due to channel error. Exiting due to channel error.
Exiting due to channel error. Exiting due to channel error.
Exiting due to channel error. Exiting due to channel error.
1552752823314 mozrunner::runner INFO Running command: "/Applications/Firefox.app/Contents/MacOS/firefox-bin" "-marionette" "-foreground" "-no-remote" "-profile" "/var/folders/bc/txjl5mmd7cb1ngc8vvs687sc0000gn/T/rust_mozprofile.3DKdFIMfEOJM"
1552752823730 addons.webextension.screenshots@mozilla.org WARN Loading extension 'screenshots@mozilla.org': Reading manifest: Invalid host permission: resource://pdf.js/
1552752823731 addons.webextension.screenshots@mozilla.org WARN Loading extension 'screenshots@mozilla.org': Reading manifest: Invalid host permission: about:reader*
Can't find symbol 'GetGraphicsResetStatus'.
1552752825108 Marionette INFO Listening on port 53822
1552752825170 Marionette WARN TLS certificate errors will be ignored for this session
1552753000086 mozrunner::runner INFO Running command: "/Applications/Firefox.app/Contents/MacOS/firefox-bin" "-marionette" "-foreground" "-no-remote" "-profile" "/var/folders/bc/txjl5mmd7cb1ngc8vvs687sc0000gn/T/rust_mozprofile.FvDFUg9EOY33"
1552753000489 addons.webextension.screenshots@mozilla.org WARN Loading extension 'screenshots@mozilla.org': Reading manifest: Invalid host permission: resource://pdf.js/
1552753000489 addons.webextension.screenshots@mozilla.org WARN Loading extension 'screenshots@mozilla.org': Reading manifest: Invalid host permission: about:reader*
Can't find symbol 'GetGraphicsResetStatus'.
1552753001869 Marionette INFO Listening on port 53925
1552753001927 Marionette WARN TLS certificate errors will be ignored for this session
[GFX1-]: Receive IPC close with reason=AbnormalShutdown
Exiting due to channel error.
Exiting due to channel error.
Exiting due to channel error.
[GFX1-]: Receive IPC close with reason=AbnormalShutdown
Exiting due to channel error.
1552753049215 mozrunner::runner INFO Running command: "/Applications/Firefox.app/Contents/MacOS/firefox-bin" "-marionette" "-foreground" "-no-remote" "-profile" "/var/folders/bc/txjl5mmd7cb1ngc8vvs687sc0000gn/T/rust_mozprofile.rpJ1QBAj04oZ"
1552753049609 addons.webextension.screenshots@mozilla.org WARN Loading extension 'screenshots@mozilla.org': Reading manifest: Invalid host permission: resource://pdf.js/
1552753049609 addons.webextension.screenshots@mozilla.org WARN Loading extension 'screenshots@mozilla.org': Reading manifest: Invalid host permission: about:reader*
Can't find symbol 'GetGraphicsResetStatus'.
1552753050997 Marionette INFO Listening on port 53983
1552753051059 Marionette WARN TLS certificate errors will be ignored for this session
1552753075848 mozrunner::runner INFO Running command: "/Applications/Firefox.app/Contents/MacOS/firefox-bin" "-marionette" "-foreground" "-no-remote" "-profile" "/var/folders/bc/txjl5mmd7cb1ngc8vvs687sc0000gn/T/rust_mozprofile.ScqclPmhfJXr"
1552753076252 addons.webextension.screenshots@mozilla.org WARN Loading extension 'screenshots@mozilla.org': Reading manifest: Invalid host permission: resource://pdf.js/
1552753076252 addons.webextension.screenshots@mozilla.org WARN Loading extension 'screenshots@mozilla.org': Reading manifest: Invalid host permission: about:reader*
Can't find symbol 'GetGraphicsResetStatus'.
1552753077641 Marionette INFO Listening on port 54037
1552753077681 Marionette WARN TLS certificate errors will be ignored for this session
[GFX1-]: Receive IPC close with reason=AbnormalShutdown
Exiting due to channel error.
[GFX1-]: Receive IPC close with reason=AbnormalShutdown
Exiting due to channel error.
[GFX1-]: Receive IPC close with reason=AbnormalShutdown
Exiting due to channel error.
[GFX1-]: Receive IPC close with reason=AbnormalShutdown
[Child 5688, Chrome_ChildThread] WARNING: pipe error: Broken pipe: file /builds/worker/workspace/build/src/ipc/chromium/src/chrome/common/ipc_channel_posix.cc, line 718
Exiting due to channel error.
1552753148688 mozrunner::runner INFO Running command: "/Applications/Firefox.app/Contents/MacOS/firefox-bin" "-marionette" "-foreground" "-no-remote" "-profile" "/var/folders/bc/txjl5mmd7cb1ngc8vvs687sc0000gn/T/rust_mozprofile.Ic9Zdp4579La"
1552753149092 addons.webextension.screenshots@mozilla.org WARN Loading extension 'screenshots@mozilla.org': Reading manifest: Invalid host permission: resource://pdf.js/
1552753149092 addons.webextension.screenshots@mozilla.org WARN Loading extension 'screenshots@mozilla.org': Reading manifest: Invalid host permission: about:reader*
Can't find symbol 'GetGraphicsResetStatus'.
1552753150486 Marionette INFO Listening on port 54098
1552753150528 Marionette WARN TLS certificate errors will be ignored for this session
console.error: BroadcastService:
receivedBroadcastMessage: handler for
remote-settings/monitor_changes
threw error:
Message: Error: Polling for changes failed: NetworkError when attempting to fetch resource..
Stack:
remoteSettingsFunction/remoteSettings.pollChanges@resource://services-settings/remote-settings.js:188:13
console.error: BroadcastService:
receivedBroadcastMessage: handler for
remote-settings/monitor_changes
threw error:
Message: Error: Polling for changes failed: NetworkError when attempting to fetch resource..
Stack:
remoteSettingsFunction/remoteSettings.pollChanges@resource://services-settings/remote-settings.js:188:13
console.error: BroadcastService:
receivedBroadcastMessage: handler for
remote-settings/monitor_changes
threw error:
Message: Error: Polling for changes failed: NetworkError when attempting to fetch resource..
Stack:
remoteSettingsFunction/remoteSettings.pollChanges@resource://services-settings/remote-settings.js:188:13
JavaScript error: https://mstdn.komittee.net/packs/js/common-0c5fef3f4c060b317d69.js, line 1: TypeError: t is undefined
JavaScript error: https://mstdn.komittee.net/packs/js/common-0c5fef3f4c060b317d69.js, line 1: TypeError: this is undefined
[GFX1-]: Receive IPC close with reason=AbnormalShutdown
[GFX1-]: Receive IPC close with reason=AbnormalShutdown
Exiting due to channel error.
Exiting due to channel error.
[GFX1-]: Receive IPC close with reason=AbnormalShutdown
[Child 5705, Chrome_ChildThread] WARNING: pipe error: Broken pipe: file /builds/worker/workspace/build/src/ipc/chromium/src/chrome/common/ipc_channel_posix.cc, line 718
Exiting due to channel error.
1552753254336 mozrunner::runner INFO Running command: "/Applications/Firefox.app/Contents/MacOS/firefox-bin" "-marionette" "-foreground" "-no-remote" "-profile" "/var/folders/bc/txjl5mmd7cb1ngc8vvs687sc0000gn/T/rust_mozprofile.G10oVxbCNtEX"
1552753254733 addons.webextension.screenshots@mozilla.org WARN Loading extension 'screenshots@mozilla.org': Reading manifest: Invalid host permission: resource://pdf.js/
1552753254733 addons.webextension.screenshots@mozilla.org WARN Loading extension 'screenshots@mozilla.org': Reading manifest: Invalid host permission: about:reader*
Can't find symbol 'GetGraphicsResetStatus'.
1552753256122 Marionette INFO Listening on port 54176
1552753256180 Marionette WARN TLS certificate errors will be ignored for this session
console.error: BroadcastService:
receivedBroadcastMessage: handler for
remote-settings/monitor_changes
threw error:
Message: Error: Polling for changes failed: NetworkError when attempting to fetch resource..
Stack:
remoteSettingsFunction/remoteSettings.pollChanges@resource://services-settings/remote-settings.js:188:13
JavaScript error: https://mstdn.komittee.net/packs/js/common-0c5fef3f4c060b317d69.js, line 1: TypeError: t is undefined
JavaScript error: https://mstdn.komittee.net/packs/js/common-0c5fef3f4c060b317d69.js, line 1: TypeError: this is undefined
[GFX1-]: Receive IPC close with reason=AbnormalShutdown
Exiting due to channel error.
[GFX1-]: Receive IPC close with reason=AbnormalShutdown
[Child 5735, Chrome_ChildThread] WARNING: pipe error: Broken pipe: file /builds/worker/workspace/build/src/ipc/chromium/src/chrome/common/ipc_channel_posix.cc, line 718
Exiting due to channel error.
[GFX1-]: Receive IPC close with reason=AbnormalShutdown
Exiting due to channel error.
[GFX1-]: Receive IPC close with reason=AbnormalShutdown
Exiting due to channel error.
1552753365874 mozrunner::runner INFO Running command: "/Applications/Firefox.app/Contents/MacOS/firefox-bin" "-marionette" "-foreground" "-no-remote" "-profile" "/var/folders/bc/txjl5mmd7cb1ngc8vvs687sc0000gn/T/rust_mozprofile.WFmSma9ncXDt"
1552753366277 addons.webextension.screenshots@mozilla.org WARN Loading extension 'screenshots@mozilla.org': Reading manifest: Invalid host permission: resource://pdf.js/
1552753366277 addons.webextension.screenshots@mozilla.org WARN Loading extension 'screenshots@mozilla.org': Reading manifest: Invalid host permission: about:reader*
Can't find symbol 'GetGraphicsResetStatus'.
1552753367658 Marionette INFO Listening on port 54249
1552753367712 Marionette WARN TLS certificate errors will be ignored for this session
console.error: BroadcastService:
receivedBroadcastMessage: handler for
remote-settings/monitor_changes
threw error:
Message: Error: Polling for changes failed: NetworkError when attempting to fetch resource..
Stack:
remoteSettingsFunction/remoteSettings.pollChanges@resource://services-settings/remote-settings.js:188:13
JavaScript error: https://mstdn.komittee.net/packs/js/common-0c5fef3f4c060b317d69.js, line 1: TypeError: t is undefined
JavaScript error: https://mstdn.komittee.net/packs/js/common-0c5fef3f4c060b317d69.js, line 1: TypeError: this is undefined
JavaScript error: https://instances.social/list#lang=en&allowed=nudity_nocw,nudity_all,spam&prohibited=&users=, line 791: TypeError: getLanguage(...) is null
1552753492306 Marionette INFO Stopped listening on port 54249
1552753493913 Marionette INFO Stopped listening on port 53983
1552753495254 Marionette INFO Stopped listening on port 53822
1552753593320 mozrunner::runner INFO Running command: "/Applications/Firefox.app/Contents/MacOS/firefox-bin" "-marionette" "-foreground" "-no-remote" "-profile" "/var/folders/bc/txjl5mmd7cb1ngc8vvs687sc0000gn/T/rust_mozprofile.tT07r7fom2UZ"
1552753593736 addons.webextension.screenshots@mozilla.org WARN Loading extension 'screenshots@mozilla.org': Reading manifest: Invalid host permission: resource://pdf.js/
1552753593736 addons.webextension.screenshots@mozilla.org WARN Loading extension 'screenshots@mozilla.org': Reading manifest: Invalid host permission: about:reader*
Can't find symbol 'GetGraphicsResetStatus'.
1552753595123 Marionette INFO Listening on port 54496
1552753595151 Marionette WARN TLS certificate errors will be ignored for this session
[GFX1-]: Receive IPC close with reason=AbnormalShutdown
Exiting due to channel error.
Exiting due to channel error.
[GFX1-]: Receive IPC close with reason=AbnormalShutdown
Exiting due to channel error.
1552753617373 mozrunner::runner INFO Running command: "/Applications/Firefox.app/Contents/MacOS/firefox-bin" "-marionette" "-foreground" "-no-remote" "-profile" "/var/folders/bc/txjl5mmd7cb1ngc8vvs687sc0000gn/T/rust_mozprofile.WBlewuR3zJKF"
1552753617776 addons.webextension.screenshots@mozilla.org WARN Loading extension 'screenshots@mozilla.org': Reading manifest: Invalid host permission: resource://pdf.js/
1552753617776 addons.webextension.screenshots@mozilla.org WARN Loading extension 'screenshots@mozilla.org': Reading manifest: Invalid host permission: about:reader*
Can't find symbol 'GetGraphicsResetStatus'.
1552753619158 Marionette INFO Listening on port 54561
1552753619215 Marionette WARN TLS certificate errors will be ignored for this session
[GFX1-]: Receive IPC close with reason=AbnormalShutdown
Exiting due to channel error.
[GFX1-]: Receive IPC close with reason=AbnormalShutdown
Exiting due to channel error.
[GFX1-]: Receive IPC close with reason=AbnormalShutdown
Exiting due to channel error.
1552753727419 mozrunner::runner INFO Running command: "/Applications/Firefox.app/Contents/MacOS/firefox-bin" "-marionette" "-foreground" "-no-remote" "-profile" "/var/folders/bc/txjl5mmd7cb1ngc8vvs687sc0000gn/T/rust_mozprofile.oG6tQi1KMh8p"
1552753727811 addons.webextension.screenshots@mozilla.org WARN Loading extension 'screenshots@mozilla.org': Reading manifest: Invalid host permission: resource://pdf.js/
1552753727811 addons.webextension.screenshots@mozilla.org WARN Loading extension 'screenshots@mozilla.org': Reading manifest: Invalid host permission: about:reader*
Can't find symbol 'GetGraphicsResetStatus'.
1552753729204 Marionette INFO Listening on port 54742
1552753729267 Marionette WARN TLS certificate errors will be ignored for this session
[GFX1-]: Receive IPC close with reason=AbnormalShutdown
Exiting due to channel error.
[GFX1-]: Receive IPC close with reason=AbnormalShutdown
[GFX1-]: Receive IPC close with reason=AbnormalShutdown
Exiting due to channel error.
Exiting due to channel error.
1552753748629 mozrunner::runner INFO Running command: "/Applications/Firefox.app/Contents/MacOS/firefox-bin" "-marionette" "-foreground" "-no-remote" "-profile" "/var/folders/bc/txjl5mmd7cb1ngc8vvs687sc0000gn/T/rust_mozprofile.TOhgY64y2G4R"
1552753749028 addons.webextension.screenshots@mozilla.org WARN Loading extension 'screenshots@mozilla.org': Reading manifest: Invalid host permission: resource://pdf.js/
1552753749028 addons.webextension.screenshots@mozilla.org WARN Loading extension 'screenshots@mozilla.org': Reading manifest: Invalid host permission: about:reader*
Can't find symbol 'GetGraphicsResetStatus'.
1552753750423 Marionette INFO Listening on port 54790
1552753750458 Marionette WARN TLS certificate errors will be ignored for this session
[GFX1-]: Receive IPC close with reason=AbnormalShutdown
Exiting due to channel error.
Exiting due to channel error.
[GFX1-]: Receive IPC close with reason=AbnormalShutdown
Exiting due to channel error.
1552753768395 mozrunner::runner INFO Running command: "/Applications/Firefox.app/Contents/MacOS/firefox-bin" "-marionette" "-foreground" "-no-remote" "-profile" "/var/folders/bc/txjl5mmd7cb1ngc8vvs687sc0000gn/T/rust_mozprofile.DMZezbeQNIie"
1552753768792 addons.webextension.screenshots@mozilla.org WARN Loading extension 'screenshots@mozilla.org': Reading manifest: Invalid host permission: resource://pdf.js/
1552753768793 addons.webextension.screenshots@mozilla.org WARN Loading extension 'screenshots@mozilla.org': Reading manifest: Invalid host permission: about:reader*
Can't find symbol 'GetGraphicsResetStatus'.
1552753770189 Marionette INFO Listening on port 54844
1552753770236 Marionette WARN TLS certificate errors will be ignored for this session
JavaScript error: https://instances.social/list#lang=en&allowed=nudity_nocw,nudity_all,spam&prohibited=&users=, line 791: TypeError: getLanguage(...) is null
JavaScript error: https://instances.social/js/bootstrap.min.js, line 6: Error: Modal is transitioning
JavaScript error: https://instances.social/js/bootstrap.min.js, line 6: Error: Modal is transitioning
JavaScript error: https://instances.social/list#lang=en&allowed=nudity_nocw,nudity_all,spam&prohibited=&users=, line 791: TypeError: getLanguage(...) is null
JavaScript error: https://instances.social/js/bootstrap.min.js, line 6: Error: Modal is transitioning
JavaScript error: https://instances.social/js/bootstrap.min.js, line 6: Error: Modal is transitioning
JavaScript error: https://instances.social/js/bootstrap.min.js, line 6: Error: Modal is transitioning
JavaScript error: https://instances.social/js/bootstrap.min.js, line 6: Error: Modal is transitioning
JavaScript error: https://instances.social/js/bootstrap.min.js, line 6: Error: Modal is transitioning
JavaScript error: https://instances.social/js/bootstrap.min.js, line 6: Error: Modal is transitioning
JavaScript error: https://instances.social/js/bootstrap.min.js, line 6: Error: Modal is transitioning
JavaScript error: https://instances.social/js/bootstrap.min.js, line 6: Error: Modal is transitioning
JavaScript error: https://instances.social/js/bootstrap.min.js, line 6: Error: Modal is transitioning
1552753827252 Marionette INFO Stopped listening on port 54844

26
selenium_instances.py
Unescape Escape View File

@ -6,6 +6,7 @@ import time
import datetime import datetime
from pprint import pprint from pprint import pprint
import requests import requests
import multiprocessing
today = datetime.date.today() today = datetime.date.today()
text_file = open("results.txt", "a+") text_file = open("results.txt", "a+")
@ -26,7 +27,6 @@ time.sleep(3)
d = 1 d = 1
i = 0 i = 0
p = -1
while True: while True:
@ -43,19 +43,16 @@ while True:
print(description.text) print(description.text)
text_file.write("Description: "+"\n"+description.text+"\n"+"\n") text_file.write("Description: "+"\n"+description.text+"\n"+"\n")
time.sleep(1) time.sleep(0.5)
# open instance in new tab # open instance in new tab
driver.find_element_by_css_selector('#modalInstanceInfo-btn-go').send_keys(Keys.COMMAND + Keys.ENTER) driver.find_element_by_css_selector('#modalInstanceInfo-btn-go').send_keys(Keys.COMMAND + Keys.ENTER)
time.sleep(2) time.sleep(0.5)
#go to new tab #go to new tab
print([p]) driver.switch_to.window(driver.window_handles[-1])
driver.switch_to.window(driver.window_handles[p])
p = p-1
time.sleep(3) time.sleep(1)
try: try:
# get the image source # get the image source
@ -66,17 +63,16 @@ while True:
if Picture_request.status_code == 200: if Picture_request.status_code == 200:
with open("image%i.jpg"%i, 'wb') as f: with open("image%i.jpg"%i, 'wb') as f:
f.write(Picture_request.content) f.write(Picture_request.content)
print("Printed Image")
except: except:
#back to original tab print("Impossible to print image")
driver.switch_to.window(driver.window_handles[0])
time.sleep(1) time.sleep(0.5)
# # NOT WORKING # close new tab
# # close new tab driver.close()
# driver.find_element_by_xpath('/html/body').send_keys(Keys.COMMAND + 'w') print("Closing Window")
# print("Closing Window")
#back to original tab #back to original tab
driver.switch_to.window(driver.window_handles[0]) driver.switch_to.window(driver.window_handles[0])

Write Preview
Loading…
Cancel
Save