CHALLENGES WITH DEBUGGING

     I am in the process of writing a script to scrape the data from the library of congress. I am using the Query language called SRU, but I have to combine it with Python for the loop. I am using a module created by an individual that allows me to use SRU with python. However, I am currently facing multiple issues associated with the module. The main error I am having is a server error which prevents me from getting any data. So far, I cannot detect the origin of the issue, but I know that this only happens during the loop. If I try to isolate the issue by making a single request with the same keywords, I do not get the error.

    With an issue like this, I can only rely on my past experience and try to debug the script in the best way possible. This is easier said than done as SRU is not so popular query language, so finding support for it is hard. I cannot rely on resources such as StackOverflow or any online forum. This is going to be the first time where I will rely mostly on myself to solve a coding problem. I will try to apply the method of handling problems I learned earlier during this internship: I will spend more time on the script, make some modifications and try to anticipate any error before running it. Once I run the program and get any unexpected exception, I will take a look at the error message, and try to isolate it by splitting the script into as many pieces as possible. However, I have been on this part of the project for too long so I have to finish it as soon as possible.

Connections with Experience

Comments

Post a Comment

Popular posts from this blog

FINAL TASK

       Today, I started what probably is my last work with Project Gutenberg. I am working on a named entity recognition. This is my second time applying timed entity recognition to this data. During my previous task, I used SpaCy to do the NER, but this time I am using JohnSnowLabs. JohnSnowLabs is a website with a collection of pre-trained machine learning pipelines. This time I am using the BERT pipeline to accomplish my task. The final result of my work will be a topic modeling and a named entity recognition which my supervisor will then analyze.      I think it is the right time to reflect on some of the things I accomplished during these nine weeks of work. I have learned a lot from this experience, whether it is a way to complete work, a learning habit, or an ability to multitask, I have gained new skills, knowledge, and work experience. I believed that these gains will allow me to perform well in my future internship.
Read more

FINAL POINT

       I recently completed my internship at Vanderbilt University, where I worked with an English professor and a librarian for a text-mining project. The goal of the project is to help the English professor in his research about how Law is represented in general dialogue during the Romanticism period (a period of literature between 1750 and 1850). So, the data collected were English books published in the United Kingdom and Ireland between 1770 and 1835. The main source of data was Project Gutenberg, an online library of free eBooks.       We faced a load of challenges while working on this project. One of the biggest issues was figuring out the correct book to collect. While project Gutenberg provides some information such as authors and editions of the book, it does not provide the year of publication, so there was no way for us to know what the right book is. We have attempted to solve the problem by collecting all the books in English and fr...
Read more