The Internet Never Forgets: A Four-Step Scraping Tutorial, Codebase, and Database for Longitudinal Organizational Website Data

IF 8.9 2区管理学 Q1 MANAGEMENT Organizational Research Methods Pub Date : 2024-11-04 DOI:10.1177/10944281241284941

Richard F.J. Haans, Marc J. Mertens

{"title":"The Internet Never Forgets: A Four-Step Scraping Tutorial, Codebase, and Database for Longitudinal Organizational Website Data","authors":"Richard F.J. Haans, Marc J. Mertens","doi":"10.1177/10944281241284941","DOIUrl":null,"url":null,"abstract":"Websites represent a crucial avenue for organizations to reach customers, attract talent, and disseminate information to stakeholders. Despite their importance, strikingly little work in the domain of organization and management research has tapped into this source of longitudinal big data. In this paper, we highlight the unique nature and profound potential of longitudinal website data and present novel open-source code- and databases that make these data accessible. Specifically, our codebase offers a general-purpose setup, building on four central steps to scrape historical websites using the Wayback Machine. Our open-access CompuCrawl database was built using this four-step approach. It contains websites of North American firms in the Compustat database between 1996 and 2020—covering 11,277 firms with 86,303 firm/year observations and 1,617,675 webpages. We describe the coverage of our database and illustrate its use by applying word-embedding models to reveal the evolving meaning of the concept of “sustainability” over time. Finally, we outline several avenues for future research enabled by our step-by-step longitudinal web scraping approach and our CompuCrawl database.","PeriodicalId":19689,"journal":{"name":"Organizational Research Methods","volume":"140 1","pages":""},"PeriodicalIF":8.9000,"publicationDate":"2024-11-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Organizational Research Methods","FirstCategoryId":"91","ListUrlMain":"https://doi.org/10.1177/10944281241284941","RegionNum":2,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"MANAGEMENT","Score":null,"Total":0}

引用次数: 0

Abstract

Websites represent a crucial avenue for organizations to reach customers, attract talent, and disseminate information to stakeholders. Despite their importance, strikingly little work in the domain of organization and management research has tapped into this source of longitudinal big data. In this paper, we highlight the unique nature and profound potential of longitudinal website data and present novel open-source code- and databases that make these data accessible. Specifically, our codebase offers a general-purpose setup, building on four central steps to scrape historical websites using the Wayback Machine. Our open-access CompuCrawl database was built using this four-step approach. It contains websites of North American firms in the Compustat database between 1996 and 2020—covering 11,277 firms with 86,303 firm/year observations and 1,617,675 webpages. We describe the coverage of our database and illustrate its use by applying word-embedding models to reveal the evolving meaning of the concept of “sustainability” over time. Finally, we outline several avenues for future research enabled by our step-by-step longitudinal web scraping approach and our CompuCrawl database.

查看原文

微信好友朋友圈 QQ好友复制链接

本刊更多论文

互联网永远不会忘记：纵向组织网站数据的四步抓取教程、代码库和数据库

网站是组织接触客户、吸引人才和向利益相关者传播信息的重要途径。尽管其重要性不言而喻，但在组织和管理研究领域，对这一纵向大数据源进行挖掘的工作却少得惊人。在本文中，我们强调了纵向网站数据的独特性质和巨大潜力，并介绍了可访问这些数据的新型开源代码和数据库。具体来说，我们的代码库提供了一种通用设置，它基于使用 Wayback Machine 搜索历史网站的四个核心步骤。我们的开放式 CompuCrawl 数据库就是采用这四个步骤建立的。它包含 Compustat 数据库中 1996 年至 2020 年北美公司的网站--涵盖 11,277 家公司，86,303 个公司/年份观察值和 1,617,675 个网页。我们介绍了数据库的覆盖范围，并通过应用词语嵌入模型来揭示 "可持续性 "概念随时间演变的含义，从而说明数据库的使用情况。最后，我们概述了利用我们的逐步纵向网络搜索方法和 CompuCrawl 数据库进行未来研究的几种途径。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文去求助

来源期刊

Organizational Research Methods Multiple-

CiteScore

23.20

自引率

3.20%

发文量

期刊介绍： Organizational Research Methods (ORM) was founded with the aim of introducing pertinent methodological advancements to researchers in organizational sciences. The objective of ORM is to promote the application of current and emerging methodologies to advance both theory and research practices. Articles are expected to be comprehensible to readers with a background consistent with the methodological and statistical training provided in contemporary organizational sciences doctoral programs. The text should be presented in a manner that facilitates accessibility. For instance, highly technical content should be placed in appendices, and authors are encouraged to include example data and computer code when relevant. Additionally, authors should explicitly outline how their contribution has the potential to advance organizational theory and research practice.