An End-to-end Web Scraping and Supervised Machine Learning Project
Jupyter Notebook
3
2,316 commits
updated Jun 26, 2023
If a person makes the Wikipedia Notable Deaths list,1 is there information there that can be used to model and predict that person's life span?
In addition to demonstrating a wide range of data science skills, I had three portfolio project criteria: (1) to scrape the data from the Web, (2) to perform extensive data cleaning (i.e., messy data), and (3) to solve a regression problem. Enter a social-sciences-based exploration of life expectancy for Wikipedia notables and we're off and running!2
"Wikipedia is a multilingual free online encyclopedia written and maintained by a community of volunteers through open collaboration and a wiki-based editing system."3 The English-language version contains a List of deaths by year of notable individuals, with links to pages for each year, by month, from 1987 to present.4 The current page format is consistent as far back as January, 1994, with the following Wikipedia-defined fields for each entry:
Name, age, country of citizenship at birth, subsequent country of citizenship (if applicable), reason for notability, cause of death (if known), and reference.5
Year, month, and day of death are also readily available, as seen in the sample in Image 1a,6 from Wikipedia Deaths in 2022, January.7
At the bottom of an indvidual's page (accessed by following the Name link), is a References section for that individual's page. Image 1b8 contains a sample from Ramiz Abutalibov's page9. The number of references is easily scraped and can represent the individual's notability, quantified. With this proxy for notability added, the above elements provide a framework for collecting the data (Image 1a-d)10 and proceeding with the analysis, as outlined in the following project overview.11
The links below access the Jupyter Notebooks that encompass the project. Standalone contents and install/run instructions introduce each notebook, with a description of the notebook's corresponding version of the dataset.

Web-scraping steps are in Notebook 1, including details of the Scrapy project folder and links to its contents.
Observations appear throughout each notebook, documenting immediate context. Exploratory Data Analysis, Linear Regression, and Modeling for Regression (Notebooks 10, 12, and 13) have additional Summary, Key Findings, or Project Recap and Conclusion sections at the end of their main content. The Project Recap and Conclusion sections are also provided below.
The link at the end of the main content of each notebook opens the next notebook. Return to README links are also available at the start and end of the main content of each notebook, to return to these instructions.
References for the entire project are below. Individual notebooks and README have self-contained footnotes.
If a person makes the Wikipedia Notable Deaths list,12 is there information there that can be used to model and predict that person's life span?
As intended, the journey took us through the processes of Web scraping, cleaning (very) messy data, and solving a regression problem.
num_references, has the 4th highest importance of predictors in the champion model.known for information (the most challenging and lengthy phase of both data cleaning and the project overall) was completely rebooted. It was at that point that the current standardized version of its code was realized. The difficult but worthwhile decision was made to redo that step with the better code.known for categories.known for categories, also part of data cleaning, was its own challenge and potential source of bias and noise. The programatically-driven manual approach to extracting this information was preferable to a purely manual approach, in that prior iterations on earlier searched columns could be easily referenced or updated for consistency and accuracy.known for at least one category other than event_record_other: an inherently noisy class that accounts for individuals known for extreme age, physical characteristics, association with or being the victim of an event, etc. To follow suit for the remaining entries, the event_record_other category was then dropped, altogether.known_for feature was engineered by combining known for categories into a single column, with 2 new classes for entries with multiple categories ("two" and "three_to_five"). The original columns would have been problematic for linear regression interpretation, due to some individuals having multiple categories.known_for feature, and (2) with the original known for categories plus num_categories (number of known for categories for an individual).We analyzed a dataset of ~78,000 entries of notable indviduals scraped from Wikipedia Notable Deaths for January, 1, 1994, through June 9, 2022,13 with the goal of ascertaining if the information there was sufficient to model a notable individual's life span. An additional ~19,400 entries were maintained separately for testing. Highlights include observed characteristics of the dataset, interpretation of key predictive features, and model performance.
known for category and num_categories predictors, and hyperparameter tuning resulted in an increase of 2.4% in explained variation in life span by the champion model, GBM2_tuned, over the linear regression model, olsmodel3.
There is scant predictive information, but not nothing.
Compared to the suggested benchmark of $R^2$ > 0.35 for machine learning models in the social sciences,15 the champion model is not a very good predictor. However, given the very narrow breadth of included predictors (region, prior region (if any), number of references (a proxy for notability), year of death, and the domain(s) for which the individual was known), explaining 11.2% of the variation in life span is reasonable.
Other potentially predictive features such as gender, marital status, income, education level, ethnicity, etc., are not overtly present in the model. It is feasible that the addition of some other key predictors to the current model's predictors (i.e., not in lieu of them) could close the gap between the model's performance and the domain's benchmark minimum $R^2$ for performance.16
Cause of death was collected but not examined in this project. COVID-19, suicide, and cancer (many types) stood out anecdotally. Comparisons of rates, both in-sample (e.g., with regard to known for category) and in relation to those of the general population, are potential areas of further study.
Delineation of known for categories is a likely source of bias. For example, in the current version of the dataset, a person who was known for activism related to an illness, who died at an early age from that illness, is included in either the politics_govt_law or social category (dependent on how their activism manifested). In many cases, the individual became an activist as a result of the diagnosis. In a sense there is a combined passive situation with resultant proactive behavior, for which the person was then known. The problem for analysis lies in the possibility that the individual is known for activity directly tied to a shortened life span. Another approach would be to add a known for category for such activism.
A source of noise is whether or not military service is captured by the law_enf_military_operator category. In particular, males of the World War II generation were very likely to have served. For those notables who were known specifically for their military service, the category is captured. For the others, it is hit or miss. In some instances, the category was captured secondary to a manual check of an individual's page for another reason. Finding the information within an individual's page is a different programmatic challenge from scraping the listed information, as we have done here.
Abid, Abubakar and Abdalla, Ali and Abid, Ali and Khan, Dawood and Alfozan, Abdulrahman and Zou, James. "Gradio: Hassle-Free Sharing and Testing of ML Models in the Wild." arXiv preprint arXiv:1906.02569. June 6, 2019. https://arxiv.org/abs/1906.02569.
Andrade, Frank. "Python Scrapy for Beginners — A Complete Web Scraping Project [Web Scraping with Python]." 2021 YouTube video, 34:17. Posted by Frank Andrade. November 9, 2021. https://www.youtube.com/watch?v=ooNngLWhTC4.
Andrade, Frank. "Web Scraping in Python BeautifulSoup, Selenium & Scrapy 2022 (Scrapy modules)." 2022 Udemy course, 98 minutes (Scrapy modules). Posted by Frank Andrade. Last modified June, 2022. https://www.udemy.com/course/web-scraping-course-in-python-bs4-selenium-and-scrapy/.
DePaul University. "Linear Correlation." Accessed November 1, 2022, https://condor.depaul.edu/sjost/it223/documents/correlation.htm.
Huizendveld, Marjin. List of nationalities. GitHub. Accessed June 17, 2022. https://gist.github.com/marijn/274449#file-nationalities-txt.
Kiprop, Victor. "A List of Nationalities." WorldAtlas. Last modified May 14, 2018. https://www.worldatlas.com/articles/what-is-a-demonym-a-list-of-nationalities.html.
Krishna and Ethan. "How to download a Jupyter Notebook from GitHub?" Stack Exchange, Data Science (blog). Last modified 21 September 2021. https://datascience.stackexchange.com/questions/35555/how-to-download-a-jupyter-notebook-from-GitHub.
Lewinson, Eryk. "Coding a custom imputer in scikit-learn." Towards Data Science. May 21, 2020. https://towardsdatascience.com/coding-a-custom-imputer-in-scikit-learn-31bd68e541de.
Nations Online Project. "Map of the World's Continents and Regions." Accessed June 29, 2022, https://www.nationsonline.org/oneworld/small_continents_map.htm.
Wikimedia Commons. "File:Screenshot snippet of Wikipedia Ramiz Abutalibov References.png." Last modified October 23, 2022. https://commons.wikimedia.org/wiki/File:Screenshot_snippet_of_Wikipedia_Ramiz_Abutalibov_References.png.
Wikimedia Commons. "File:Wikipedia Deaths Jan 2022 snippet.png." Last modified October 23,2022. https://commons.wikimedia.org/wiki/File:Wikipedia_Deaths_Jan_2022_snippet.png.
Wikipedia. "Deaths in 2022." Last modified October 24, 2022. https://en.wikipedia.org/wiki/Deaths_in_2022.
Wikipedia. "Deaths in January 1994" through "Deaths in June 2022" (through June 9, 2022) and each listed individual's page. Accessed (scraped) June 9-10, 2022, https://en.wikipedia.org/wiki/Deaths_in_January_1994.
Wikipedia. "Deaths in January 2022." Last modified October 24, 2022. https://en.wikipedia.org/wiki/Deaths_in_January_2022.
Wikipedia. "Lists of deaths by year." Last modified October 1, 2022. https://en.wikipedia.org/wiki/Lists_of_deaths_by_year.
Wikipedia. "Ramiz Abutalibov." Last modified May 14, 2022. https://en.wikipedia.org/wiki/Ramiz_Abutalibov.
Wikipedia. "Sudhakar Chaturvedi." Last modified October, 24, 2022. https://en.wikipedia.org/wiki/Sudhakar_Chaturvedi.
Wikipedia. "Wikipedia." Last modified October 20, 2022. https://en.wikipedia.org/wiki/Wikipedia.
The overall approach to coding in Python, analysis and modeling, and a majority of the plots (Notebooks 10, 11, 12, and 13, and Project Overview images above) are adapted from examples learned in The University of Texas McCombs School of Business Post Graduate Program in Data Science and Business Analytics in partnership with Great Learning.
Git and GitHub implementation was acquired from Anna Skoulikari, through the Udemy course, Git Learning Journey - Guide to Learn Git (Version Control).
Review of Jupyter Notebooks for pull requests on GitHub was entirely dependent on ReviewNB, following Amit Rathi's instructions in the Towards Data Science article, "How to use Git / GitHub with Jupyter Notebook".
Web scraping with Scrapy was learned from Frank Andrade, through the Udemy course, Web Scraping in Python BeautifulSoup, Selenium & Scrapy 2022.
Though not extensively used in this project, SQL implementation was acquired from Jose Portilla, through the Udemy course, The Complete SQL Bootcamp 2022: Go from Zero to Hero. The steps for implementing SQLite with pandas are from Alan Jones' Towards Data Science article, "Python Pandas and SQLite".
Custom imputer coding steps are from Eryk Lewinson's Towards Data Science article, "Coding a custom imputer in scikit-learn", with the addition of a strategy for mode.
The instructions for Gradio Blocks were a starting point for the Gradio demo, combined with following the class documentation (i.e., Shift + Tab). To have the Gradio demo permanently hosted on Hugging Face Spaces, these instructions were followed. The files are accessible in Files and versions on the space. Note that the requirements.txt file was necessary for the app to successfully build on the Hugging Face Space. The app.py file is a copy and paste of Notebook 14 code, directly into the browser, as prompted when creating the space.
Tableau implementation was learned from Kirill Eremenko and the Ligency Team, through the Udemy course, Tableau 2022 A-Z: Hands-On Tableau Training for Data Science.
Thank you to all of the Wikipedians and Wikipedia Notables, whose respective contributions and fascinating lives lived made this exploration possible.
This project serves as an end-to-end portfolio project for its author, as sole contributor.
wikipedia-life-expectancy (Text and Data) © 2022 by Teresa Hanak is licensed under CC BY-SA 4.0
wikipedia-life-expectancy (Code) released under MIT License
Copyright 2022 Teresa Hanak
2,316 commits
Jupyter Notebook
99.9%
An End-to-end Web Scraping and Supervised Machine Learning Project
Jupyter Notebook
3
2,316 commits
updated Jun 26, 2023
If a person makes the Wikipedia Notable Deaths list,1 is there information there that can be used to model and predict that person's life span?
In addition to demonstrating a wide range of data science skills, I had three portfolio project criteria: (1) to scrape the data from the Web, (2) to perform extensive data cleaning (i.e., messy data), and (3) to solve a regression problem. Enter a social-sciences-based exploration of life expectancy for Wikipedia notables and we're off and running!2
"Wikipedia is a multilingual free online encyclopedia written and maintained by a community of volunteers through open collaboration and a wiki-based editing system."3 The English-language version contains a List of deaths by year of notable individuals, with links to pages for each year, by month, from 1987 to present.4 The current page format is consistent as far back as January, 1994, with the following Wikipedia-defined fields for each entry:
Name, age, country of citizenship at birth, subsequent country of citizenship (if applicable), reason for notability, cause of death (if known), and reference.5
Year, month, and day of death are also readily available, as seen in the sample in Image 1a,6 from Wikipedia Deaths in 2022, January.7
At the bottom of an indvidual's page (accessed by following the Name link), is a References section for that individual's page. Image 1b8 contains a sample from Ramiz Abutalibov's page9. The number of references is easily scraped and can represent the individual's notability, quantified. With this proxy for notability added, the above elements provide a framework for collecting the data (Image 1a-d)10 and proceeding with the analysis, as outlined in the following project overview.11
The links below access the Jupyter Notebooks that encompass the project. Standalone contents and install/run instructions introduce each notebook, with a description of the notebook's corresponding version of the dataset.

Web-scraping steps are in Notebook 1, including details of the Scrapy project folder and links to its contents.
Observations appear throughout each notebook, documenting immediate context. Exploratory Data Analysis, Linear Regression, and Modeling for Regression (Notebooks 10, 12, and 13) have additional Summary, Key Findings, or Project Recap and Conclusion sections at the end of their main content. The Project Recap and Conclusion sections are also provided below.
The link at the end of the main content of each notebook opens the next notebook. Return to README links are also available at the start and end of the main content of each notebook, to return to these instructions.
References for the entire project are below. Individual notebooks and README have self-contained footnotes.
If a person makes the Wikipedia Notable Deaths list,12 is there information there that can be used to model and predict that person's life span?
As intended, the journey took us through the processes of Web scraping, cleaning (very) messy data, and solving a regression problem.
num_references, has the 4th highest importance of predictors in the champion model.known for information (the most challenging and lengthy phase of both data cleaning and the project overall) was completely rebooted. It was at that point that the current standardized version of its code was realized. The difficult but worthwhile decision was made to redo that step with the better code.known for categories.known for categories, also part of data cleaning, was its own challenge and potential source of bias and noise. The programatically-driven manual approach to extracting this information was preferable to a purely manual approach, in that prior iterations on earlier searched columns could be easily referenced or updated for consistency and accuracy.known for at least one category other than event_record_other: an inherently noisy class that accounts for individuals known for extreme age, physical characteristics, association with or being the victim of an event, etc. To follow suit for the remaining entries, the event_record_other category was then dropped, altogether.known_for feature was engineered by combining known for categories into a single column, with 2 new classes for entries with multiple categories ("two" and "three_to_five"). The original columns would have been problematic for linear regression interpretation, due to some individuals having multiple categories.known_for feature, and (2) with the original known for categories plus num_categories (number of known for categories for an individual).We analyzed a dataset of ~78,000 entries of notable indviduals scraped from Wikipedia Notable Deaths for January, 1, 1994, through June 9, 2022,13 with the goal of ascertaining if the information there was sufficient to model a notable individual's life span. An additional ~19,400 entries were maintained separately for testing. Highlights include observed characteristics of the dataset, interpretation of key predictive features, and model performance.
known for category and num_categories predictors, and hyperparameter tuning resulted in an increase of 2.4% in explained variation in life span by the champion model, GBM2_tuned, over the linear regression model, olsmodel3.
There is scant predictive information, but not nothing.
Compared to the suggested benchmark of $R^2$ > 0.35 for machine learning models in the social sciences,15 the champion model is not a very good predictor. However, given the very narrow breadth of included predictors (region, prior region (if any), number of references (a proxy for notability), year of death, and the domain(s) for which the individual was known), explaining 11.2% of the variation in life span is reasonable.
Other potentially predictive features such as gender, marital status, income, education level, ethnicity, etc., are not overtly present in the model. It is feasible that the addition of some other key predictors to the current model's predictors (i.e., not in lieu of them) could close the gap between the model's performance and the domain's benchmark minimum $R^2$ for performance.16
Cause of death was collected but not examined in this project. COVID-19, suicide, and cancer (many types) stood out anecdotally. Comparisons of rates, both in-sample (e.g., with regard to known for category) and in relation to those of the general population, are potential areas of further study.
Delineation of known for categories is a likely source of bias. For example, in the current version of the dataset, a person who was known for activism related to an illness, who died at an early age from that illness, is included in either the politics_govt_law or social category (dependent on how their activism manifested). In many cases, the individual became an activist as a result of the diagnosis. In a sense there is a combined passive situation with resultant proactive behavior, for which the person was then known. The problem for analysis lies in the possibility that the individual is known for activity directly tied to a shortened life span. Another approach would be to add a known for category for such activism.
A source of noise is whether or not military service is captured by the law_enf_military_operator category. In particular, males of the World War II generation were very likely to have served. For those notables who were known specifically for their military service, the category is captured. For the others, it is hit or miss. In some instances, the category was captured secondary to a manual check of an individual's page for another reason. Finding the information within an individual's page is a different programmatic challenge from scraping the listed information, as we have done here.
Abid, Abubakar and Abdalla, Ali and Abid, Ali and Khan, Dawood and Alfozan, Abdulrahman and Zou, James. "Gradio: Hassle-Free Sharing and Testing of ML Models in the Wild." arXiv preprint arXiv:1906.02569. June 6, 2019. https://arxiv.org/abs/1906.02569.
Andrade, Frank. "Python Scrapy for Beginners — A Complete Web Scraping Project [Web Scraping with Python]." 2021 YouTube video, 34:17. Posted by Frank Andrade. November 9, 2021. https://www.youtube.com/watch?v=ooNngLWhTC4.
Andrade, Frank. "Web Scraping in Python BeautifulSoup, Selenium & Scrapy 2022 (Scrapy modules)." 2022 Udemy course, 98 minutes (Scrapy modules). Posted by Frank Andrade. Last modified June, 2022. https://www.udemy.com/course/web-scraping-course-in-python-bs4-selenium-and-scrapy/.
DePaul University. "Linear Correlation." Accessed November 1, 2022, https://condor.depaul.edu/sjost/it223/documents/correlation.htm.
Huizendveld, Marjin. List of nationalities. GitHub. Accessed June 17, 2022. https://gist.github.com/marijn/274449#file-nationalities-txt.
Kiprop, Victor. "A List of Nationalities." WorldAtlas. Last modified May 14, 2018. https://www.worldatlas.com/articles/what-is-a-demonym-a-list-of-nationalities.html.
Krishna and Ethan. "How to download a Jupyter Notebook from GitHub?" Stack Exchange, Data Science (blog). Last modified 21 September 2021. https://datascience.stackexchange.com/questions/35555/how-to-download-a-jupyter-notebook-from-GitHub.
Lewinson, Eryk. "Coding a custom imputer in scikit-learn." Towards Data Science. May 21, 2020. https://towardsdatascience.com/coding-a-custom-imputer-in-scikit-learn-31bd68e541de.
Nations Online Project. "Map of the World's Continents and Regions." Accessed June 29, 2022, https://www.nationsonline.org/oneworld/small_continents_map.htm.
Wikimedia Commons. "File:Screenshot snippet of Wikipedia Ramiz Abutalibov References.png." Last modified October 23, 2022. https://commons.wikimedia.org/wiki/File:Screenshot_snippet_of_Wikipedia_Ramiz_Abutalibov_References.png.
Wikimedia Commons. "File:Wikipedia Deaths Jan 2022 snippet.png." Last modified October 23,2022. https://commons.wikimedia.org/wiki/File:Wikipedia_Deaths_Jan_2022_snippet.png.
Wikipedia. "Deaths in 2022." Last modified October 24, 2022. https://en.wikipedia.org/wiki/Deaths_in_2022.
Wikipedia. "Deaths in January 1994" through "Deaths in June 2022" (through June 9, 2022) and each listed individual's page. Accessed (scraped) June 9-10, 2022, https://en.wikipedia.org/wiki/Deaths_in_January_1994.
Wikipedia. "Deaths in January 2022." Last modified October 24, 2022. https://en.wikipedia.org/wiki/Deaths_in_January_2022.
Wikipedia. "Lists of deaths by year." Last modified October 1, 2022. https://en.wikipedia.org/wiki/Lists_of_deaths_by_year.
Wikipedia. "Ramiz Abutalibov." Last modified May 14, 2022. https://en.wikipedia.org/wiki/Ramiz_Abutalibov.
Wikipedia. "Sudhakar Chaturvedi." Last modified October, 24, 2022. https://en.wikipedia.org/wiki/Sudhakar_Chaturvedi.
Wikipedia. "Wikipedia." Last modified October 20, 2022. https://en.wikipedia.org/wiki/Wikipedia.
The overall approach to coding in Python, analysis and modeling, and a majority of the plots (Notebooks 10, 11, 12, and 13, and Project Overview images above) are adapted from examples learned in The University of Texas McCombs School of Business Post Graduate Program in Data Science and Business Analytics in partnership with Great Learning.
Git and GitHub implementation was acquired from Anna Skoulikari, through the Udemy course, Git Learning Journey - Guide to Learn Git (Version Control).
Review of Jupyter Notebooks for pull requests on GitHub was entirely dependent on ReviewNB, following Amit Rathi's instructions in the Towards Data Science article, "How to use Git / GitHub with Jupyter Notebook".
Web scraping with Scrapy was learned from Frank Andrade, through the Udemy course, Web Scraping in Python BeautifulSoup, Selenium & Scrapy 2022.
Though not extensively used in this project, SQL implementation was acquired from Jose Portilla, through the Udemy course, The Complete SQL Bootcamp 2022: Go from Zero to Hero. The steps for implementing SQLite with pandas are from Alan Jones' Towards Data Science article, "Python Pandas and SQLite".
Custom imputer coding steps are from Eryk Lewinson's Towards Data Science article, "Coding a custom imputer in scikit-learn", with the addition of a strategy for mode.
The instructions for Gradio Blocks were a starting point for the Gradio demo, combined with following the class documentation (i.e., Shift + Tab). To have the Gradio demo permanently hosted on Hugging Face Spaces, these instructions were followed. The files are accessible in Files and versions on the space. Note that the requirements.txt file was necessary for the app to successfully build on the Hugging Face Space. The app.py file is a copy and paste of Notebook 14 code, directly into the browser, as prompted when creating the space.
Tableau implementation was learned from Kirill Eremenko and the Ligency Team, through the Udemy course, Tableau 2022 A-Z: Hands-On Tableau Training for Data Science.
Thank you to all of the Wikipedians and Wikipedia Notables, whose respective contributions and fascinating lives lived made this exploration possible.
This project serves as an end-to-end portfolio project for its author, as sole contributor.
wikipedia-life-expectancy (Text and Data) © 2022 by Teresa Hanak is licensed under CC BY-SA 4.0
wikipedia-life-expectancy (Code) released under MIT License
Copyright 2022 Teresa Hanak
2,316 commits
Jupyter Notebook
99.9%