PROGRESS IN DATA COLLECTION

 The data collection process

    Today, I had my first meeting with the other fellow students and professors. During the meeting, I had a chance to chat with a professor about the work I am doing and the stage where I currently am at. The professor told me about his interest in the same data and how unsuccessful he was so far. I got to share with him some of the methods I have used, my result so far, and the new method I am planning to try. This is now my third straight blog post talking about the same step, which is collecting books published in the U.K. from a specific period of time. This is a very tricky part of the project as there is no good way to collect the data. Project Gutenberg has the books, but not their country or year of publication, so I only selected the books that are written in English, then I have to rely on other sources to figure out when and where each of the books was published. 

    After exploring all the possible options, I am now planning to try WorldCat. WorldCat has an API but it is paid and is not easily accessible to an individual. So, instead of using an API, I will scrape the website with a python script using a website emulator Selenium Webdriver. This method is not guaranteed to work 100% as I do expect to be denied access to the website after a certain amount of requests. This issue can be solved by putting a short timeout between each request or a longer one between a batch of requests. This timeout will minimize any impact such a large query would have on the WorldCat server. 

    Chatting with that professor today made me realize that other people are actually looking for the same data that we are, and yet, it is not easily accessible. So, I am planning to collect the year and country of publication of all the books in English from Project Gutenberg and store the data somewhere online for people to use.

Comments

Post a Comment

Popular posts from this blog

FINAL TASK

       Today, I started what probably is my last work with Project Gutenberg. I am working on a named entity recognition. This is my second time applying timed entity recognition to this data. During my previous task, I used SpaCy to do the NER, but this time I am using JohnSnowLabs. JohnSnowLabs is a website with a collection of pre-trained machine learning pipelines. This time I am using the BERT pipeline to accomplish my task. The final result of my work will be a topic modeling and a named entity recognition which my supervisor will then analyze.      I think it is the right time to reflect on some of the things I accomplished during these nine weeks of work. I have learned a lot from this experience, whether it is a way to complete work, a learning habit, or an ability to multitask, I have gained new skills, knowledge, and work experience. I believed that these gains will allow me to perform well in my future internship.
Read more

FINAL POINT

       I recently completed my internship at Vanderbilt University, where I worked with an English professor and a librarian for a text-mining project. The goal of the project is to help the English professor in his research about how Law is represented in general dialogue during the Romanticism period (a period of literature between 1750 and 1850). So, the data collected were English books published in the United Kingdom and Ireland between 1770 and 1835. The main source of data was Project Gutenberg, an online library of free eBooks.       We faced a load of challenges while working on this project. One of the biggest issues was figuring out the correct book to collect. While project Gutenberg provides some information such as authors and editions of the book, it does not provide the year of publication, so there was no way for us to know what the right book is. We have attempted to solve the problem by collecting all the books in English and fr...
Read more