BACK TO SQUARE ONE

When terms and conditions are meant to be read

    After completing a fully-working version of the Worldcat crawler and presenting it to my supervisor, I was informed that my script could not be used and that I have to come up with another method to collect the data. As a reminder, I found out that WorldCat has most of the data that we are looking for. It does have an API but it is paid and can only be accessed by an organization or a company. So, I decided to crawl the website instead to extract the data we need. Today, I presented the result to my supervisor and was informed that WorldCat's policies prevent us from using it. WorldCat mentions in its terms and conditions that scraping, crawling, and bot-accessing the website is prohibited. My supervisor had to inform me about this for me to realize it. 

    Before this research project, I have scraped many websites to collect their data for personal projects and school projects, so never considered any legal issues as I was not commercializing or sharing any of the data, and my queries were too small to have any effect on the website server. So, what happened to me was a total beginner mistake. Even if I could not use my script, the time I spent working on it was not a total waste as I learned a new thing. I learned a way to tackle a new problem, anticipate possible problems, and solve unexpected problems effectively. I will apply these methods to my current internship as a whole: I will take more time to plan ahead for every work I need to complete. I will explore every possible way by using methods such as pseudocode, graphs, or diagrams. I will spend more time exploring any possible issue related to each of the methods I selected, this time, from the perspective of an organization, not an individual doing a personal project. This step will allow me to anticipate any possible issue and will allow me to make the right choice. I will be as organized as possible in all my work, it will allow me to track down the root of any unexpected issue effectively. I know, these measures will not guarantee that I will not make any beginner mistakes again in the future, but these measures will decrease the number of mistakes and errors considerably.

Topic: Connections to experiences

Comments

Post a Comment

Popular posts from this blog

FINAL TASK

       Today, I started what probably is my last work with Project Gutenberg. I am working on a named entity recognition. This is my second time applying timed entity recognition to this data. During my previous task, I used SpaCy to do the NER, but this time I am using JohnSnowLabs. JohnSnowLabs is a website with a collection of pre-trained machine learning pipelines. This time I am using the BERT pipeline to accomplish my task. The final result of my work will be a topic modeling and a named entity recognition which my supervisor will then analyze.      I think it is the right time to reflect on some of the things I accomplished during these nine weeks of work. I have learned a lot from this experience, whether it is a way to complete work, a learning habit, or an ability to multitask, I have gained new skills, knowledge, and work experience. I believed that these gains will allow me to perform well in my future internship.
Read more

FINAL POINT

       I recently completed my internship at Vanderbilt University, where I worked with an English professor and a librarian for a text-mining project. The goal of the project is to help the English professor in his research about how Law is represented in general dialogue during the Romanticism period (a period of literature between 1750 and 1850). So, the data collected were English books published in the United Kingdom and Ireland between 1770 and 1835. The main source of data was Project Gutenberg, an online library of free eBooks.       We faced a load of challenges while working on this project. One of the biggest issues was figuring out the correct book to collect. While project Gutenberg provides some information such as authors and editions of the book, it does not provide the year of publication, so there was no way for us to know what the right book is. We have attempted to solve the problem by collecting all the books in English and fr...
Read more