DATA PROCESSING

     Today, I manipulated data that I would consider fairly big for the first time. It is the collection of all Project Gutenberg books, which has a total size of 31GB. I am supposed to process the books in order to create metadata of Project Gutenberg, including the full text, title, and author. When I wrote the code, I wrote it fully and debugged it on the go whenever an issue occurs. This is how I usually work but it was not the best method this time. Due to the large amount of data, it takes several hours for a task to complete, so running everything and then stopping midway to debug was time-consuming. Sometimes, my script ran for hours before I realize that something does not work fine, so I had to stop the script, edit the code, then ran it from the beginning again.

    It took me about a week to complete the task, and I am still running the final version of the code while writing the prompt. It made me realize that with such data, it is much more efficient to do a ground truth with a small sample of the data. The ground truth is the process of running the code with a small portion of data, where the script can run and finish within a short period. This method makes debugging much easier and takes less time. The concept of ground truth is that when everything works fine, we simply change the small data to the bigger one, and the code should perform the same. I have learned this the hard way, and I will make sure to not make this mistake again in the future.

Comments

Post a Comment

Popular posts from this blog

FINAL TASK

       Today, I started what probably is my last work with Project Gutenberg. I am working on a named entity recognition. This is my second time applying timed entity recognition to this data. During my previous task, I used SpaCy to do the NER, but this time I am using JohnSnowLabs. JohnSnowLabs is a website with a collection of pre-trained machine learning pipelines. This time I am using the BERT pipeline to accomplish my task. The final result of my work will be a topic modeling and a named entity recognition which my supervisor will then analyze.      I think it is the right time to reflect on some of the things I accomplished during these nine weeks of work. I have learned a lot from this experience, whether it is a way to complete work, a learning habit, or an ability to multitask, I have gained new skills, knowledge, and work experience. I believed that these gains will allow me to perform well in my future internship.
Read more

FINAL POINT

       I recently completed my internship at Vanderbilt University, where I worked with an English professor and a librarian for a text-mining project. The goal of the project is to help the English professor in his research about how Law is represented in general dialogue during the Romanticism period (a period of literature between 1750 and 1850). So, the data collected were English books published in the United Kingdom and Ireland between 1770 and 1835. The main source of data was Project Gutenberg, an online library of free eBooks.       We faced a load of challenges while working on this project. One of the biggest issues was figuring out the correct book to collect. While project Gutenberg provides some information such as authors and editions of the book, it does not provide the year of publication, so there was no way for us to know what the right book is. We have attempted to solve the problem by collecting all the books in English and fr...
Read more