You explored Python programming and some capabilities of NLTK in tutorials 1 and 2, and in assignment 1. As you
are aware, NLTK is a free set of Python libraries and resources for natural language processing. It contains
implementations of NLP algorithms, access to over 50 corpora, and lexical resources a WordNet. The NLP libraries
include libraries for classification, tokenization, stemming, tagging, parsing, and semantic analysis [1]. In this
assignment, you will explore the support available for n-gram language implementation in NLTK and implement
and use a n-gram language model. Then you will document the process you followed in creating the n-gram
language model.
Tasks
1. Refer the NLTK Language Modeling Module of the NLTK documentation [3] and get familiar with how to
implement a n-gram language model using NLTK. You may use the tutorials [4], [5], and [6] as additional
resources. Also, you are free to use any other learning resource.
2. Load the Brown corpus from the resources available in [2]. Pick sentences from three categories in the
corpus and use them as the dataset for your language model.
a) Note that sentences in this corpus have period at the end and you may have to remove them as
a preprocessing task.
3. Divide the set of sentences into two groups, 90% of sentences into the training set and remaining 10%
into the testing set.
COMPUTER INFORMATION SYSTEMS
COMP 482: NATURAL LANGUAGE PROCESSING
COMP 482 – Assignment 2 Page 2 of 3
4. Create a bi-gram language model and, train it using the training set.
a) You will have to pad the sentences with start of the sentence and end of the sentence symbols
before using them. Make sure you do the same processing to the testing set.
b) You are not required to use smoothing for this assignment. However, if you wish you may use
smoothing.
c) You are required to handle unknown words.
5. Calculate t
he probability score for three random sentences using your language model.
a) You are required to include screenshots of the calculation (with the sentence shown) for this
step in your report.
6. Create three random sentences using your language model.
a) You are required to include screenshots with the commands used to generate the sentences
and the generated sentences in your report.
7. Test the quality of your generated model using the testing set and report using appropriate measures that
are used to evaluate language models.
a) You are required to include screenshots with the commands used for evaluation and results
obtained in your report.
8. Compile a report indicating the tasks you performed. Your report must contain following items and can
contain any other material you think is important.
a) Explanation of how you built the model. This section should be a comprehensive explanation of
how the language model was created, used and evaluated. This section should be written
entirely on your own words and if any material is taken from other sources should be properly
cited. In this section, you have to show/convince me that you know the process of creating a
language model, using it, and evaluating it. This is the most important section of your work. Any
plagiarism/copying detected in this section will result in you getting zero marks for the entire
assignment.
b) Screenshots from Step 5.
c) Screenshots from Step 6.
d) Screenshots from Step 7.
e) Your comments and observations about the created language model.
9. Submit a single zip file which contains the following. The zip file should be named .zip (for example <300321321.zip), where .zip is the extension.
a) Source code for tasks 2 to 7 in a python source (.py) file. Make sure your code is commented.
b) Any other resources or material required to execute your code.
c) Your report in PDF format