ONLINE RESOURCE VS OWN SCRIPT

     On Wednesday, I was tasked to create a metadata of the entire Project Gutenberg corpus, containing the full book text, the author, and the title. This metadata will be saved online, we will get our portion of data from it, and it may also be useful to other researchers in the future. I found a project on Github that attempted to collect the Project Gutenberg books and process them, but the code is broken in many ways such as it only collects half of the data, it only processes a very small portion of the data, and its filter (per language) does not work. Due to this, the program is practically useless, and I have to collect the data manually. Due to the large size of the data and to prevent the PG's database is not saturated, there is a small delay after each request. Due to this, it takes a few days to collect a total of 68.000 books. I am now on my third day of download and I only got 46.000 books. 

    It made me realize that although there are many resources online, these are not always reliable, and instead of using that resource, it is better to try to understand how it works and reproduce it. Then I found out that trying to debug a code of someone else is much harder than creating your own program, especially if the code is not commented. I have always been told to comment on a code, and, since I always know what I do, I do not feel the need to put comments, but this project made me realize how important comment is, not for me, but for anyone who will use my program in the future.

Comments

Post a Comment

Popular posts from this blog

FINAL TASK

       Today, I started what probably is my last work with Project Gutenberg. I am working on a named entity recognition. This is my second time applying timed entity recognition to this data. During my previous task, I used SpaCy to do the NER, but this time I am using JohnSnowLabs. JohnSnowLabs is a website with a collection of pre-trained machine learning pipelines. This time I am using the BERT pipeline to accomplish my task. The final result of my work will be a topic modeling and a named entity recognition which my supervisor will then analyze.      I think it is the right time to reflect on some of the things I accomplished during these nine weeks of work. I have learned a lot from this experience, whether it is a way to complete work, a learning habit, or an ability to multitask, I have gained new skills, knowledge, and work experience. I believed that these gains will allow me to perform well in my future internship.
Read more

FINAL POINT

       I recently completed my internship at Vanderbilt University, where I worked with an English professor and a librarian for a text-mining project. The goal of the project is to help the English professor in his research about how Law is represented in general dialogue during the Romanticism period (a period of literature between 1750 and 1850). So, the data collected were English books published in the United Kingdom and Ireland between 1770 and 1835. The main source of data was Project Gutenberg, an online library of free eBooks.       We faced a load of challenges while working on this project. One of the biggest issues was figuring out the correct book to collect. While project Gutenberg provides some information such as authors and editions of the book, it does not provide the year of publication, so there was no way for us to know what the right book is. We have attempted to solve the problem by collecting all the books in English and fr...
Read more