MANUAL VS AUTOMATED

    This week, I focused on storing all the texts that we got from Project Gutenberg on a single CSV file, along with their title and author. After completing everything, I reviewed the file and found out that I was missing around ninety books. I had two options to solve these issues: I can either re-run the entire program from scratch including the book download or search each of the books manually from the Project Gutenberg website and copy the text. The first option seems to be a much better option because everything is automated. Also, if I rerun it, it checks if the book is already downloaded before it downloads it, so it only gets the missing books. But there are two reasons why this option is not viable: downloading the books only takes about 2, and running everything does not guarantee success because the reason why the books were not downloaded was not identified. Therefore, the only option was to search each of the books manually. During the process, I found out that around twenty-four books were actually audio files, so have to be discarded. the other ones were text files so I copied and pasted each of them manually.

   That part task was "the 20% that takes 80% of the time", and it taught me to always consider every single option and compare each of their advantages and drawbacks before choosing the one to choose. Sometimes the options that one thing may be the worst may be the most suitable for a specific task. I applied this new skill to a programming task, but I am planning to also apply this as my decision-making method in general. The task may have been manual but I finished it hours earlier than the automated option and I could debug the code which leads to the completion of the task way before the deadline and a much better quality of output that satisfies my mentors.

Comments

Post a Comment

Popular posts from this blog

FINAL TASK

       Today, I started what probably is my last work with Project Gutenberg. I am working on a named entity recognition. This is my second time applying timed entity recognition to this data. During my previous task, I used SpaCy to do the NER, but this time I am using JohnSnowLabs. JohnSnowLabs is a website with a collection of pre-trained machine learning pipelines. This time I am using the BERT pipeline to accomplish my task. The final result of my work will be a topic modeling and a named entity recognition which my supervisor will then analyze.      I think it is the right time to reflect on some of the things I accomplished during these nine weeks of work. I have learned a lot from this experience, whether it is a way to complete work, a learning habit, or an ability to multitask, I have gained new skills, knowledge, and work experience. I believed that these gains will allow me to perform well in my future internship.
Read more

FINAL POINT

       I recently completed my internship at Vanderbilt University, where I worked with an English professor and a librarian for a text-mining project. The goal of the project is to help the English professor in his research about how Law is represented in general dialogue during the Romanticism period (a period of literature between 1750 and 1850). So, the data collected were English books published in the United Kingdom and Ireland between 1770 and 1835. The main source of data was Project Gutenberg, an online library of free eBooks.       We faced a load of challenges while working on this project. One of the biggest issues was figuring out the correct book to collect. While project Gutenberg provides some information such as authors and editions of the book, it does not provide the year of publication, so there was no way for us to know what the right book is. We have attempted to solve the problem by collecting all the books in English and fr...
Read more