Automated Grading of Python in VPL using Unit Tests

A few of my colleagues here at YorkU have expressed an interest in using Python with Virtual Programming Lab on our Moodle-based eClass Learning Management System. I don't have a lot of Python experience, having only really used it for some data processing and toy embedded system demonstrations in the past.

So I adapted the Unit Test examples that I did earlier this summer for C (blog posts 1, 2, 3, 4 and 5; original source D. Thiébaut) and had Bing / ChatGPT write up a sample function and a unit testing Python program for it. I then adapted it for VPL and the result is here. I'm going to leave out important details like:

How to set up a VPL exercise on eClass
How to point the VPL exercise to our local VPL server

as I've discussed those elsewhere and I just want to get the main scripts and Python files listed here. Again, I used ChatGPT to create the Python files, so I don't claim any authorship there. I'm sure that someone with actual Python experience can do a better job and will use the baseline here to get started.

So, what's important here is to have two Python files:

primaryFile.py
- This is what the teacher runs to test the student's submission. It contains the unit test and outputs "PASS" or "FAIL" that get captured by the VPL script files.
StudentSubmission.py
- This is the file that the student is supposed to write. Here, it just adds to input parameters and outputs the result (c = a+b)

In addition, we have two scripts in VPL:

vpl_run.sh
- This script will run primaryFile.py and will let the student know if the test passed or failed. No grade recorded.
vpl_evaluate.sh
- This script does the same as vpl_run.sh but will record a grade (1 or 0) and send it to the eClass grade book.

Video Overview

Here is a video overview of how to set up this Python example in VPL.

Video on how to set up Python with Unit Tests in VPL. https://youtu.be/oRcdxmyogpE

Files

Here are the four important files for creating this example in VPL.

primaryFile.py

This is what the teacher runs to test the student's submission. It contains the unit test and outputs "PASS" or "FAIL" that get captured by the VPL script files.

# generated in Bing / ChatGPT
# Import the unittest module
import unittest

# Import the StudentSubmission.py file
import StudentSubmission

# Define a class that inherits from unittest.TestCase
class TestAdd(unittest.TestCase):

    # Define a method that starts with test_
    def test_add(self):
        # Use self.assertEqual to compare the expected and actual results
        self.assertEqual(StudentSubmission.add(1, 2), 3) # Pass
        self.assertEqual(StudentSubmission.add(-1, 1), 0) # Pass
        self.assertEqual(StudentSubmission.add(2, 2), 4) # Pass

# Run the unit test
if __name__ == '__main__':
    # Create a test suite
    suite = unittest.TestSuite()
    # Add the test method to the suite
    suite.addTest(TestAdd('test_add'))
    # Create a test runner
    runner = unittest.TextTestRunner()
    # Run the test suite and capture the result
    result = runner.run(suite)
    # Check if the result was successful
    if result.wasSuccessful():
        # Output the word pass
        print("PASS")
    else:
        # Output the word fail
        print("FAIL")

StudentSubmission.py

Put this in "requested files" and pretend the student submitted it.

# Generated in ChatGPT / Bing.
# Define a function named add that takes two numbers as parameters and returns their sum
def add(a, b):
    # Add the two numbers and store the result in a variable named c
    c = a + b
    # Return the value of c
    return c

vpl_run.sh

This is the first script that tests the student code. It uses the rocket button in VPL. No grade is assigned. It will show if the student code in StudentSubmission.py passes or fails the unit tests.

#!/bin/bash
#
# vpl_run.sh script 
# An approach for Unit Tested Python programs in VPL, based on similar work for C.
# Uses bash script and looks for PASS or FAIL from the unit test. 
#
# 
# James Andrew Smith; drsmith@yorku.ca July 2021 (updated August 2023)
#
#
# Based on an approach to unit testing for C (July & August 2023)
# 1. https://www.yorku.ca/professor/drsmith/2023/07/31/automated-student-evaluations-in-c-part-1/
# 2. https://www.yorku.ca/professor/drsmith/2023/08/19/automated-student-evaluations-in-c-part-2/
# 3. https://www.yorku.ca/professor/drsmith/2023/08/19/automated-student-evaluations-in-c-part-3/
# 4. https://www.yorku.ca/professor/drsmith/2023/08/19/automated-student-evaluations-in-c-part-4/
# 5. https://www.yorku.ca/professor/drsmith/2023/08/19/automated-student-evaluations-in-c-part-5/
#
# That C work was based on example by D. Thiébaut at Smith College
# http://www.science.smith.edu/dftwiki/index.php/Tutorial:_Moodle_VPL_--_Testing_a_C_Program
# now at
# https://web.archive.org/web/20190208191644/http://www.science.smith.edu/dftwiki/index.php/Tutorial:_Moodle_VPL_--_Testing_a_C_Program
#
# August 2023 update: removed the SED whitespace processing as (1) it messes with
# the ASCII art from the flowchart and (2) it's not necessary with the intoduction
# of the Unity unit tester.
# -----------------------------------------------------------------


cat > vpl_execution <<EEOOFF
#!/bin/bash

# the number of times we need to see a unit test pass.
NUM_PASSES=1    # TO-DO : update this (in vpl_run and _evaluate) 

# output file from execution of the main program
OUTFILE="user.out"

# Variables (don't add whitespace)
maxGrade=1
minGrade=0
consolationGrade=0
 
# Set the minimum grade.  (slashes due to this script writing vpl_executable)
grade=\$( expr \$minGrade )

# --- program tested  ---
# 
python3 primaryFile.py &> \$OUTFILE

# If the file contains the string "PASS" a certain number of times (set above)
# but also check to see that "FAIL" is not returned.
if [ \$(grep -o "PASS" \$OUTFILE | wc -l) -eq \$NUM_PASSES ]; then
    # Check if the file does not contain the string "FAIL"
    if ! grep -q "FAIL" \$OUTFILE; then
        echo "The test on your code succeeded."
    else
        echo "The test on your code did _not_ succeed."
    fi
else
    echo "The test on your code did _not_ succeed."
fi

echo "***********************"
echo "The test on the student's code returned the following message:"
cat \$OUTFILE
echo "***********************"

echo "***********************"
echo "No grade has been saved."
echo "Use the evaluation button for grading."
echo "***********************"

EEOOFF

 
chmod +x vpl_execution

vpl_evaluate.sh

This is the script that students execute by clicking on the checkmark icon. It will submit a grade (0 or 1) to eClass based on whether the student's submission met the requirements of the unit tests.

#!/bin/bash
#
# vpl_run.sh script 
# An approach for Unit Tested Python programs in VPL, based on similar work for C.
# Uses bash script and looks for PASS or FAIL from the unit test. 
#
# 
# James Andrew Smith; drsmith@yorku.ca July 2021 (updated August 2023)
#
#
# Based on an approach to unit testing for C (July & August 2023)
# 1. https://www.yorku.ca/professor/drsmith/2023/07/31/automated-student-evaluations-in-c-part-1/
# 2. https://www.yorku.ca/professor/drsmith/2023/08/19/automated-student-evaluations-in-c-part-2/
# 3. https://www.yorku.ca/professor/drsmith/2023/08/19/automated-student-evaluations-in-c-part-3/
# 4. https://www.yorku.ca/professor/drsmith/2023/08/19/automated-student-evaluations-in-c-part-4/
# 5. https://www.yorku.ca/professor/drsmith/2023/08/19/automated-student-evaluations-in-c-part-5/
#
# That C work was based on example by D. Thiébaut at Smith College
# http://www.science.smith.edu/dftwiki/index.php/Tutorial:_Moodle_VPL_--_Testing_a_C_Program
# now at
# https://web.archive.org/web/20190208191644/http://www.science.smith.edu/dftwiki/index.php/Tutorial:_Moodle_VPL_--_Testing_a_C_Program
#
# August 2023 update: removed the SED whitespace processing as (1) it messes with
# the ASCII art from the flowchart and (2) it's not necessary with the intoduction
# of the Unity unit tester.
# -----------------------------------------------------------------


cat > vpl_execution <<EEOOFF
#!/bin/bash

# the number of times we need to see a unit test pass.
NUM_PASSES=1    # TO-DO : update this (in vpl_run and _evalute) 

# output file from execution of the main program
OUTFILE="user.out"

# Variables (don't add whitespace)
maxGrade=1
minGrade=0
consolationGrade=0
 
# Set the minimum grade.  (slashes due to this script writing vpl_executable)
grade=\$( expr \$minGrade )

# --- program tested (no extension) ---
# 
python3 primaryFile.py &> \$OUTFILE

# If the file contains the string "PASS" a certain number of times (set above)
# but also check to see that "FAIL" is not returned.
if [ \$(grep -o "PASS" \$OUTFILE | wc -l) -eq \$NUM_PASSES ]; then
    # Check if the file does not contain the string "FAIL"
    if ! grep -q "FAIL" \$OUTFILE; then
      echo "Comment :=>>- Congratulations: your function is CORRECT."
      echo "Comment :=>> -----------------"
      echo "Comment :=>>- The result of the testing program, with the student function included:"
      echo "<|--"
      cat user.out
      echo ""
      echo "--|>"
      grade=\$( expr \$maxGrade )
    else
      echo "Comment :=>>- Your program is NOT correct. Fix it."
      echo "Comment :=>> ---------------"
      echo "Comment :=>>- The result of the testing program, with the student function included:"
      echo "Comment :=>> ---------------"
      echo "<|--"
      cat user.out
      grade=\$( expr \$minGrade )
    fi
else
      echo "Comment :=>>- Your program is NOT correct. Fix it."
      echo "Comment :=>> ---------------"
      echo "Comment :=>>- The result of the testing program, with the student function included:"
      echo "Comment :=>> ---------------"
      echo "<|--"
      cat user.out
      grade=\$( expr \$minGrade )
fi
   

# Send the grade to eClass via a special-formatted comment.
echo "The following grade is being sent to eClass: \$grade"
echo "Grade :=>> \$grade"

EEOOFF

 
chmod +x vpl_execution

Edits:

November 15, 2023: added the YouTube video and some support text.

James Andrew Smith is a Professional Engineer and Associate Professor in the Electrical Engineering and Computer Science Department of York University's Lassonde School, with degrees in Electrical and Mechanical Engineering from the University of Alberta and McGill University. Previously a program director in biomedical engineering, his research background spans robotics, locomotion, human birth and engineering education. While on sabbatical in 2018-19 with his wife and kids he lived in Strasbourg, France and he taught at the INSA Strasbourg and Hochschule Karlsruhe and wrote about his personal and professional perspectives. James is a proponent of using social media to advocate for justice, equity, diversity and inclusion as well as evidence-based applications of research in the public sphere. You can find him on Twitter. Originally from Québec City, he now lives in Toronto, Canada.