onebigear
/
special-issue-11-wiki2html
forked from XPUB/special-issue-11-wiki2html
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
You cannot select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
86 Commits
9 Branches
0 Tags
5.9 MiB
HTML 40.8%
Python 33.2%
CSS 24.6%
Shell 1%
Jinja 0.2%
Other 0.2%
 
 
 
 
 
 
master
ezn
itempage
populat_orgs
docker
pdf2jpg
sgolub
mark
andre
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from 'e5bcb3ab91'
${ noResults }
Go to file
Castro0o e5bcb3ab91 generating document html pages with images in 4 columns rows 5 years ago
examples example of ask as the outer loop 5 years ago
sandbox playing wiht Jinja 5 years ago
static generating document html pages with images in 4 columns rows 5 years ago
templates generating document html pages with images in 4 columns rows 5 years ago
.gitignore example of ask as the outer loop 5 years ago
README.md test push2 5 years ago
download_imgs.py downloading images with clean filenames and resizing:jpg,jpeg,png 5 years ago
example_api_calls.py example calling mediawiki api via mwclient to make thumbnails and then to get semantic data 5 years ago
functions.py todos 5 years ago
helper-upload_imgs_dir.sh Missing backslash 5 years ago
imgs_info.py images2html 5 years ago
pdf2jpg.sh pdf2jpg.sh ready 5 years ago
query2html.py generating document html pages with images in 4 columns rows 5 years ago
run.sh change name of script publication2html.py --> ask2html.py 5 years ago
upload_imgs_dir.py sh/py 5 years ago

README.md

Wiki to HTML pages script

Depencencies

  • python3

  • pip Python library installed

    • Install:
      • curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
      • python3 get-pip.py

  • mwclient Python library

    • Install:
      • pip3 install mwclient

  • jinja2 Python library

    • Install:
      • pip3 install jinja2

  • Pillow Python library for image processing

    • pip3 install Pillow

  • pandoc

    • Install:
      • Debian/Ubuntu: sudo apt install pandoc
      • Mac: brew install pandoc

login.txt

login.txt is a local and individual file, ignored by git, where you place you itch wiki username and password, in separate lines.

It is used to let mwclient access the wiki, since it is close for reading and writing.

myusername
mypassword

Run

cd special-issue-11-wiki2html/

Run scripts together with ./run.sh

1 script at a time:

python3 download_imgs.py

  • Downloads all images from wiki to images/ directory
  • and stores each image's metadata to images.json

python3 query2html.py

  • with ask API perform a query:
    • help python3 query2html.py --help
    • run dry python3 query2html.py --dry only printing request, not executing it
    • build custom query with arguments --conditions --printouts --sort --order
    • default query is: [[File:+]][[Title::+]][[Part::+]][[Date::+]]|?Title|?Date|?Part|?Partof|sort=Date,Title,Part|order=asc,asc,asc
    • custom queries
      • python3 query2html.py --conditions '[[Date::>=1970/01/01]][[Date::<=1979/12/31]]'
      • python3 query2html.py --conditions '[[Creator::~*task force*]]'

Note: to avoid confusion or problems is better to leave the --printouts --sort --order arguments as the default. Otherwise document parts will start to get grouped not according to their Title, hence creating documents made from different original parts.

How does query2html.py work?

Based on the query made: MW API will send back a number of Page titles that match the query conditions, together with its printouts (metadata proprety::value pairs).

For each Page:

  • its locally stored image is found
  • its text retrieved from MW
  • a fragment of html (document_part_html) is generated based on the templates/document_part.html

All Pages that share the same metadata's Title value, will:

  • gather all their html fragments in all_document_parts
  • render templates/document.html with the content of all_document_parts
  • save the render template to 'static_html/DocumentTitle.html',

Each of the saved documents:

  • render templates/index.html with the info on each document has been saved into documentslist
  • resulting in static_html/index.html

Bulk image upload upload_imgs_dir.py

Get Help: python3 upload_imgs_dir.py --help

Edit and run via .helper-upload_imgs_dir.sh

Convert PDFs to folder of JPGs with pdf2jpg.sh

By either:

  • running it from this folder and using absolute path to PDF ./pdf2jpg.sh "/full/path/to/2020_bantayog/PDFname.pdf"

  • copying pdf2jpg.sh to 2020_bantayog/ and running with relative path to PDF ./pdf2jpg.sh "PDFname.pdf"

It is

to convert pdfs to jpgs: convert -quality 100 -density 300 [name-of-pdf] %02d.jpg