You cannot select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
465 lines
11 KiB
Plaintext
465 lines
11 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"# MediaWiki API (Dérive)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"This notebook:\n",
|
|
"\n",
|
|
"* continues with exploring the connections between `Hypertext` & `Dérive`\n",
|
|
"* saves (parts of) wiki pages as html files"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"import urllib\n",
|
|
"import json\n",
|
|
"from IPython.display import JSON # iPython JSON renderer\n",
|
|
"import sys"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Parse"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Let's use another wiki this time: the English Wikipedia.\n",
|
|
"\n",
|
|
"You can pick any page, i took the Hypertext page for this notebook as an example: https://en.wikipedia.org/wiki/Hypertext"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# parse the wiki page Hypertext\n",
|
|
"request = 'https://en.wikipedia.org/w/api.php?action=parse&page=Hypertext&format=json'\n",
|
|
"response = urllib.request.urlopen(request).read()\n",
|
|
"data = json.loads(response)\n",
|
|
"JSON(data)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": []
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": []
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Wiki links dérive\n",
|
|
"\n",
|
|
"Select the wiki links from the `data` response:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"scrolled": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"links = data['parse']['links']\n",
|
|
"JSON(links)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Let's save the links as a list of pagenames, to make it look like this:\n",
|
|
"\n",
|
|
"`['hyperdocuments', 'hyperwords', 'hyperworld']`"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"scrolled": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# How is \"links\" structured now?\n",
|
|
"print(links)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"It helps to copy paste a small part of the output first:\n",
|
|
"\n",
|
|
"`[{'ns': 0, 'exists': '', '*': 'Metatext'}, {'ns': 0, '*': 'De man met de hoed'}]`\n",
|
|
"\n",
|
|
"and to write it differently with indentation:\n",
|
|
"\n",
|
|
"```\n",
|
|
"links = [\n",
|
|
" { \n",
|
|
" 'ns' : 0,\n",
|
|
" 'exists' : '',\n",
|
|
" '*', 'Metatext'\n",
|
|
" }, \n",
|
|
" {\n",
|
|
" 'ns' : 0,\n",
|
|
" 'exists' : '',\n",
|
|
" '*' : 'De man met de hoed'\n",
|
|
" } \n",
|
|
"]\n",
|
|
"```\n",
|
|
"\n",
|
|
"We can now loop through \"links\" and add all the pagenames to a new list called \"wikilinks\"."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"scrolled": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"wikilinks = []\n",
|
|
"\n",
|
|
"for link in links:\n",
|
|
" \n",
|
|
" print('link:', link)\n",
|
|
" \n",
|
|
" for key, value in link.items():\n",
|
|
" print('----- key:', key)\n",
|
|
" print('----- value:', value)\n",
|
|
" print('-----')\n",
|
|
" \n",
|
|
" pagename = link['*']\n",
|
|
" print('===== pagename:', pagename)\n",
|
|
" \n",
|
|
" wikilinks.append(pagename)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"scrolled": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"wikilinks"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": []
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Saving the links in a HTML page\n",
|
|
"\n",
|
|
"Let's convert the list of pagenames into HTML link elements (`<a href=\"\"></a>`):"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"scrolled": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"html = ''\n",
|
|
"\n",
|
|
"for wikilink in wikilinks:\n",
|
|
" print(wikilink)\n",
|
|
" \n",
|
|
" # let's use the \"safe\" pagenames for the filenames \n",
|
|
" # by replacing the ' ' with '_'\n",
|
|
" filename = wikilink.replace(' ', '_')\n",
|
|
" \n",
|
|
" a = f'<a href=\"{ filename }.html\">{ wikilink }</a>'\n",
|
|
" html += a\n",
|
|
" html += '\\n'"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"scrolled": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"print(html)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": []
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Let's save this page in a separate folder, i called it \"mediawiki-api-dérive\"\n",
|
|
"# We can make this folder here using a terminal command, but you can also do it in the interface on the left\n",
|
|
"! mkdir mediawiki-api-dérive"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"output = open('mediawiki-api-dérive/Hypertext.html', 'w')\n",
|
|
"output.write(html)\n",
|
|
"output.close()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": []
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": []
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Recursive parsing\n",
|
|
"\n",
|
|
"We can now repeat the steps for each wikilink that we collected!\n",
|
|
"\n",
|
|
"We can make an API request for each wikilink, \\\n",
|
|
"ask for all the links on the page \\\n",
|
|
"and save it as an HTML page."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# First we save the Hypertext page again:\n",
|
|
"\n",
|
|
"startpage = 'Hypertext'\n",
|
|
"\n",
|
|
"# parse the first wiki page\n",
|
|
"request = f'https://en.wikipedia.org/w/api.php?action=parse&page={ startpage }&format=json'\n",
|
|
"response = urllib.request.urlopen(request).read()\n",
|
|
"data = json.loads(response)\n",
|
|
"JSON(data)\n",
|
|
"\n",
|
|
"# select the links\n",
|
|
"links = data['parse']['links']\n",
|
|
"\n",
|
|
"# turn it into a list of pagenames\n",
|
|
"wikilinks = []\n",
|
|
"for link in links:\n",
|
|
" pagename = link['*']\n",
|
|
" wikilinks.append(pagename)\n",
|
|
"\n",
|
|
"# turn the wikilinks into a set of <a href=\"\"></a> links\n",
|
|
"html = ''\n",
|
|
"for wikilink in wikilinks:\n",
|
|
" filename = wikilink.replace(' ', '_')\n",
|
|
" a = f'<a href=\"{ filename }.html\">{ wikilink }</a>'\n",
|
|
" html += a\n",
|
|
" html += '\\n'\n",
|
|
"\n",
|
|
"# save it as a HTML page\n",
|
|
"startpage = startpage.replace(' ', '_') # let's again stay safe on the filename side\n",
|
|
"output = open(f'mediawiki-api-dérive/{ startpage }.html', 'w')\n",
|
|
"output.write(html)\n",
|
|
"output.close()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Then we loop through the list of wikilinks\n",
|
|
"# and repeat the steps for each page\n",
|
|
" \n",
|
|
"for wikilink in wikilinks:\n",
|
|
" \n",
|
|
" # let's copy the current wikilink pagename, to avoid confusion later\n",
|
|
" currentwikilink = wikilink \n",
|
|
" print('Now requesting:', currentwikilink)\n",
|
|
" \n",
|
|
" # parse this wiki page\n",
|
|
" wikilink = wikilink.replace(' ', '_')\n",
|
|
" request = f'https://en.wikipedia.org/w/api.php?action=parse&page={ wikilink }&format=json'\n",
|
|
" \n",
|
|
" # --> we insert a \"try and error\" condition, \n",
|
|
" # to catch errors in case a page does not exist \n",
|
|
" try: \n",
|
|
" \n",
|
|
" # continue the parse request\n",
|
|
" response = urllib.request.urlopen(request).read()\n",
|
|
" data = json.loads(response)\n",
|
|
" JSON(data)\n",
|
|
"\n",
|
|
" # select the links\n",
|
|
" links = data['parse']['links']\n",
|
|
"\n",
|
|
" # turn it into a list of pagenames\n",
|
|
" wikilinks = []\n",
|
|
" for link in links:\n",
|
|
" pagename = link['*']\n",
|
|
" wikilinks.append(pagename)\n",
|
|
"\n",
|
|
" # turn the wikilinks into a set of <a href=\"\"></a> links\n",
|
|
" html = ''\n",
|
|
" for wikilink in wikilinks:\n",
|
|
" filename = wikilink.replace(' ', '_')\n",
|
|
" a = f'<a href=\"{ filename }.html\">{ wikilink }</a>'\n",
|
|
" html += a\n",
|
|
" html += '\\n'\n",
|
|
"\n",
|
|
" # save it as a HTML page\n",
|
|
" currentwikilink = currentwikilink.replace(' ', '_') # let's again stay safe on the filename side\n",
|
|
" output = open(f'mediawiki-api-dérive/{ currentwikilink }.html', 'w')\n",
|
|
" output.write(html)\n",
|
|
" output.close()\n",
|
|
" \n",
|
|
" except:\n",
|
|
" error = sys.exc_info()[0]\n",
|
|
" print('Skipped:', wikilink)\n",
|
|
" print('With the error:', error)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": []
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## What's next?\n",
|
|
"\n",
|
|
"?\n",
|
|
"\n",
|
|
"You could add more loops to the recursive parsing, adding more layers ...\n",
|
|
"\n",
|
|
"You could request all images of a page (instead of links) ...\n",
|
|
"\n",
|
|
"or something else the API offers ... (contributors, text, etc)\n",
|
|
"\n",
|
|
"or ..."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": []
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": []
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": []
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": []
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": []
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "Python 3",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.7.3"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 4
|
|
}
|