SCRAPING WORLDCAT

 


The final step for getting the books' publication dates

    Yesterday, I started writing the script to scrape WorldCat using Python and Selenium. This task is particularly taught and time-consuming. I spent more than 15 hours working on it and it is still not finished. The trickiest part is that I am trying to get the date each of the +17 thousand books was published. My method is that I pass the title of a book to the search bar, go through the results, find the earliest edition, then extract the publication date. The tricky part is that I am working with string data (both the "titles" and "authors" are strings), but I have to make sure that I get the right book from the right author from the search result. Unlike an integer data type, a lot of variations may be seen in a string making it hard to make an exact comparison. Therefore, it is much harder to generalize the script. 

    The best option I had to create this script was by creating a much simpler version, running it, detecting any issue or exception, then trying to solve the exception in a more general way, then running it again. When the small version works fine, I add one or two more features, then repeat the process. At the time of writing this post, I am at the step where I print out the date for the earliest edition of the books. I experienced an exception in the way the website is laid out, so I will have to fix it. My current script is that I go to a page that contains the list of all the available editions to find the earliest one. But there are some books that only have one edition so the page I am supposed to access does not exist, hence I get an error. So, I will create a condition to tackle this exception.

    While thinking about the new condition I need to create, I realized that I could improve the way I feed the data into the search box: Instead of only searching for the title of the book and then finding the one by the correct author from the result, I can, instead, feed in the "title by author" which will only return the result with the best match. I will still need a few more days to complete this script. It is likely that I will be able to create a perfect general script that satisfies all the 17 thousand books, but I will surely create one that will allow me to collect the maximum data possible.

Comments

Post a Comment

Popular posts from this blog

FINAL TASK

       Today, I started what probably is my last work with Project Gutenberg. I am working on a named entity recognition. This is my second time applying timed entity recognition to this data. During my previous task, I used SpaCy to do the NER, but this time I am using JohnSnowLabs. JohnSnowLabs is a website with a collection of pre-trained machine learning pipelines. This time I am using the BERT pipeline to accomplish my task. The final result of my work will be a topic modeling and a named entity recognition which my supervisor will then analyze.      I think it is the right time to reflect on some of the things I accomplished during these nine weeks of work. I have learned a lot from this experience, whether it is a way to complete work, a learning habit, or an ability to multitask, I have gained new skills, knowledge, and work experience. I believed that these gains will allow me to perform well in my future internship.
Read more

FINAL POINT

       I recently completed my internship at Vanderbilt University, where I worked with an English professor and a librarian for a text-mining project. The goal of the project is to help the English professor in his research about how Law is represented in general dialogue during the Romanticism period (a period of literature between 1750 and 1850). So, the data collected were English books published in the United Kingdom and Ireland between 1770 and 1835. The main source of data was Project Gutenberg, an online library of free eBooks.       We faced a load of challenges while working on this project. One of the biggest issues was figuring out the correct book to collect. While project Gutenberg provides some information such as authors and editions of the book, it does not provide the year of publication, so there was no way for us to know what the right book is. We have attempted to solve the problem by collecting all the books in English and fr...
Read more