TEAM WORK

     NLTK is arguably one of the best tools out there for removing stopwords. It is widely used and well supported, so I decided to use it for our project. I realized that while NLTK did its job well, we still ended up with a lot of stopwords that confused our model and ruined the result of our topic modeling. The reason for this was due to the fact that we are working with 2-century-old data. There are different types of words that have changed over time, some words that are proper to a specific country/region, like Scotland for example. There are words such as "otter" or "weel". These words are not recognized by NLTK so could not be removed. So, one of my supervisors, an English professor, who is also the head of the project, had to make a list of all these other stopwords manually and gave them to me so that I could add them to the code.

    It is a prime example of how it is always better to have different people with specific expertise on a specific subject when working on a project. This is what makes working on an individual project is very difficult. I realized this as I am planning to create a text generator in my native language Malagasy. I know the coding and NLP part, but I am very bad at grammar. Thankfully, my mother teaches the language at a high school, so cooperating with her by collecting any grammar data I may need from her will make it more realistic to realize my project.

    We are also starting a data science group on campus and the co-leads are from different departments such as business, biology, and math. we will all contribute to a project where each student will contribute with their expertise in their specific domain.

Comments

Post a Comment

Popular posts from this blog

FINAL TASK

       Today, I started what probably is my last work with Project Gutenberg. I am working on a named entity recognition. This is my second time applying timed entity recognition to this data. During my previous task, I used SpaCy to do the NER, but this time I am using JohnSnowLabs. JohnSnowLabs is a website with a collection of pre-trained machine learning pipelines. This time I am using the BERT pipeline to accomplish my task. The final result of my work will be a topic modeling and a named entity recognition which my supervisor will then analyze.      I think it is the right time to reflect on some of the things I accomplished during these nine weeks of work. I have learned a lot from this experience, whether it is a way to complete work, a learning habit, or an ability to multitask, I have gained new skills, knowledge, and work experience. I believed that these gains will allow me to perform well in my future internship.
Read more

FINAL POINT

       I recently completed my internship at Vanderbilt University, where I worked with an English professor and a librarian for a text-mining project. The goal of the project is to help the English professor in his research about how Law is represented in general dialogue during the Romanticism period (a period of literature between 1750 and 1850). So, the data collected were English books published in the United Kingdom and Ireland between 1770 and 1835. The main source of data was Project Gutenberg, an online library of free eBooks.       We faced a load of challenges while working on this project. One of the biggest issues was figuring out the correct book to collect. While project Gutenberg provides some information such as authors and editions of the book, it does not provide the year of publication, so there was no way for us to know what the right book is. We have attempted to solve the problem by collecting all the books in English and fr...
Read more