SQL vs PYTHON

    The new data collection process was fruitful compared to the previous ones. This latest approach made me realize that the main goal is to get the books from a particular era, not to get the publication date of the book. As a reminder, our latest method was based on a list of authors that one of my supervisors provided. The list is a metadata containing a list of multiple authors and the period during which they published their work. The period we are looking for is the Romanticism period from 1780 to 1837. So I started from with that list, filtered the authors who published their works during the period we are looking for. we create a new metadata for these authors. Then we take the new metadata of the authors and the project gutenberg metadata, then we join the them based on the name of authors. In the end we end up with a new metadata containing the British authors and the books they have published during the romantic period.

    All the task described above was done via SQL query, a language that I used for the first time during this project. I started this internship with a very limited knowledge of SQL, and despite the book that I read and the tutorials I have watched, I have never really used SQL in a project before this one. So, I had to learn a lot while doing the work, I had to google pretty much any idea I had in mind to find out if SQL had it. I was surprised to see that the SQL syntax shares some similarities with Python, a language that I use a lot. The similarities mostly occur in the conditions where both languages use "and" and "or". I assume that these similarities made it easier for me to learn SQL and complete work with it despite my lack of prior experience.

Comments

Post a Comment

Popular posts from this blog

FINAL TASK

       Today, I started what probably is my last work with Project Gutenberg. I am working on a named entity recognition. This is my second time applying timed entity recognition to this data. During my previous task, I used SpaCy to do the NER, but this time I am using JohnSnowLabs. JohnSnowLabs is a website with a collection of pre-trained machine learning pipelines. This time I am using the BERT pipeline to accomplish my task. The final result of my work will be a topic modeling and a named entity recognition which my supervisor will then analyze.      I think it is the right time to reflect on some of the things I accomplished during these nine weeks of work. I have learned a lot from this experience, whether it is a way to complete work, a learning habit, or an ability to multitask, I have gained new skills, knowledge, and work experience. I believed that these gains will allow me to perform well in my future internship.
Read more

FINAL POINT

       I recently completed my internship at Vanderbilt University, where I worked with an English professor and a librarian for a text-mining project. The goal of the project is to help the English professor in his research about how Law is represented in general dialogue during the Romanticism period (a period of literature between 1750 and 1850). So, the data collected were English books published in the United Kingdom and Ireland between 1770 and 1835. The main source of data was Project Gutenberg, an online library of free eBooks.       We faced a load of challenges while working on this project. One of the biggest issues was figuring out the correct book to collect. While project Gutenberg provides some information such as authors and editions of the book, it does not provide the year of publication, so there was no way for us to know what the right book is. We have attempted to solve the problem by collecting all the books in English and fr...
Read more