Micro-Feedback

 Data and Assessing Courses

When we teach, either online courses or in the classroom, gathering information for assessment is critical to improve what we do as educators.  Assessment is one of the aspects that educators often don’t do well.  Many times, course evaluation surveys are the extent of our attempts to assess whether we are meeting the courses goals. These surveys are distributed at the end of the course and if the instructor is lucky, the results may be available months later.  This process neglects the needs of the instructor by not affording them the information they need to make corrections and improve in real time.  Naturally, the students also suffer, as they will not benefit from real time corrections by the instructor.

The ADDIE model is a traditional instructional design methodology to help streamline the production of your course.  The name is an acronym for the 5 steps of the model:

1. Analyze
2. Design
3. Develop
4. Implement
5. Evaluation

Evaluation is set at the end of the process, and while the process is supposed to be cyclical, often critiques of the process is that it is too time consuming.  Models like Rapid Instruction Design (RID) or Agile emphasize including evaluation throughout the stage.  Whether these critiques of the ADDIE model or if traditionally people have failed to apply the ADDIE model functionally at different levels is an interesting question, but will not directly help you improve you course now.

Micro-Feedback

Developing mechanisms to collect feedback throughout the course, or micro-feedback, and using it to improve your course on the fly can be very helpful.  By including short surveys at the module level (or smaller), instead of just one large survey at the end of the course, instructors can gain valuable insight that can help them rapidly adapt to their students’ needs.  This improvement will directly affect the students. Moreover, the adaptation can meet the specific needs of the students ad may vary the next time the course is taught.

Adopting a flexible and adaptive approach using micro-feedback gives you a greater understanding of your course.  Surveys can be short, and can include some reflective feedback from the student to give you the information you need to both better meet their needs and improve student success.  While it may seem difficult, adding a small (5 question) option survey can give you valuable insight on improving your courses.  You may find that the information can help you by giving you insight that could streamline your instruction and reduce your work by making it more effective.

AI and Computer Programming

Computer Science faculty have been dealing with AI tools and coding before the vast popularity of generative AI (Gen AI) in 2023. Since that popularity, there has been much attention to how AI will transform software development and coding.

Pros of AI Assisted Programming

Specifically, proponents of Gen AI identify key benefits that it offers. These include:

 

• Decrease coding time: Programmers in code up to twice as fast using generative AI (Deniz et al, 2023).
• Higher Satisfaction: 60-75% of programmers have reported that using AI felt more fulfilled with their job. Likewise, 73% stated that AI assisted their focus and 87% stated that they benefited from the reduced mental effort form repetitive tasks (Kalliamvakou,2024). The increase rate of adopting the suggestions from AI also correlates with the programmers perceived satisfaction (Ziegler et al, 2022).

 
However, it is important to note that perceived satisfaction from surveys may contain biases that is not the same as quantitative research on productivity and programmer focus (Moore, 2024b)

Cons of AI Assisted Programming

However, there are also significant costs that come with using gen AI to code. A few of these costs include:

• Security: Programmers who use AI for assisting coding are more likely to introduce security vulnerabilities in their code (Perry et al, 2023). Those investing in repeated inquiries and examining the AI generated code could off-set this increase, but at a cost of time saved by using AI.
• Productivity: There has been no productivity gain and potentially some serious downsides. First, programmers using gen AI had a 41% increase in their bug rate (Moore, 2024a). This creates a greater need to debug the code created.
• Satisfaction and Burn-out: Controlled for sustained effort, the extended worktime outside of standard times shows that there was no reduction of the rate of burn-out from programmers when using AI. (Moore, 2024c)

These costs suggest that whatever perceived benefits of using Ai to code can be off-set by the increased time debugging and examining the code it generates.

The appealing nature of AI coding tools to create vast amounts of code poses long-term danger: as developers get accustomed to the perceived speed of the AI tools, they may gain false confidence and rely on them to program and review code more. This will perpetrate more bugs and will be amplified by code that is not documented. There will also be a skill loss in organizations.

Tips When Using AI Assisted Programming

If one is going to use Gen AI to assist with programming, it is important to understand its limitations. This entails:

• Repeatedly examine the code for bugs and errors. Gen AI repeatedly produces code with errors. These errors do not reduce when increasing its prompts. In some cases, AI produced twice as many errors and took longer to patch errors (Jesse, 2023).
• Do not rely on Gen AI for the most difficult programming issues. AI increasingly creates unsatisfactory code the harder the problem becomes. This is also the case when working with specific organizational content. Since the AI was not trained on the specific content, it regularly creates errors that prefer its training data and not solving the problem in within the organizational context (Moore, 2024b).

Given these needs, programmers need to leverage their skills to monitor code to reduce the error rates. Producing less bugs and security issues in the final product will be the hallmark of desired programmers. This will also leverage programming expertise against the speed of dubious code produced by Gen AI and ensure that skilled programmers will be more preferable than those that rely on Gen AI to produce their final product.

How Does that Affect Education?

If decisions are not made soon, educators will lose the ability to shape future programmers and face future challenges to come (Becker, et al, 2023). Responsible educators must remind their students of critical issues associated with coding with AI. Further they need to produce learners who can assess and reflect on Gen AI as a tool instead of a solution.

References

Becker, B, Denny, James, P, Finnie-Ansley, J, Luxton-Reilly, A., Prather, J, and E. Antonio Santos (2023). Programming Is Hard - Or at Least It Used to Be: Educational Opportunities and Challenges of AI Code Generation. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 (SIGCSE 2023). Association for Computing Machinery, New York, NY, USA, 500–506.

Cui, Z, Demirer, M, Jaffe, S, Musolff, L, Peng, S, and T. Salz (2025) The Effects of Generative AI on High-Skilled Work: Evidence from Three Field Experiments with Software Developers Feb 10

Deniz B.K. Gnanasambandam, C, Harrysson, M, Hussin, A and S. Srivastava (2023) Unleashing developer productivity with generative AI. McKinsey Digital. June.

Jesse, K, Ahmed, T, Devanbu, P, & E. Morgan. (2023). Language Models and Simple, Stupid Bugs. Zenodo.  Feb. 25.  

Kalliamvakou, E (2024) Research: quantifying GitHub Copilot’s impact on developer productivity and happiness. GitHub.

Meyer, A, Fritz, T, Murphy, G, and T. Zimmermann. (2014). Software developers' perceptions of productivity. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE 2014). Association for Computing Machinery, New York, NY, USA, 19–29.

Moore, N (2024a) AI won’t solve your developer productivity problems for you, yet. Uplevel. Oct 18.

Moore, N (2024b) Improving Developer Experience Requires Quantitative Data. Uplevel, Jul 5.

Moore, N (2024c) Developer Toil: How to Stop It from Burning Out Your Team. Uplevel. July 26.

Perry, N, Srivatava, M, Kumar, D, and D, Boneh (2023) Do Users Write More Insecure Code with AI Assistants?. In Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security (CCS ’23), November 26–30, 2023, Copenhagen, Denmark. ACM, New York, NY, USA.

Ziegler, A, Kalliamvakou, E, Li, A, Rice,A, Rifkin,D, Simister, S, Sittampalam, G, and E.Aftandilian. (2022) Productivity Assessment of Neural Code Completion. In Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming (MAPS ’22), June 13, 2022, San Diego, CA, USA. ACM, New York, NY, USA

AI in Education – Chat-GPT and a Serious Concern

While there has been considerable attention recently about Artificial Intelligence, AI, development in the field is not new. In 1956 Dartmouth college coined the term during a summer research project organized by Dr. John McCarthy.


Since then, “artificial intelligence” has been used in one way or another in education for years. Search engines, personal assistants on phones, assistive technology to increase accessibility, and other technology all use some form of applied artificial intelligence. However, what most people are concerned about is the advances OpenAI technology found in GPT-4 and ChatGPT. To address this, let's briefly look at how ChatGPT works and a few of the issues they pose through their architecture.

CHAT-GPT: How it Works

  Unlike Deep Blue, that uses “brute force” to check all possible outcomes to determine an optimal answer, ChatGPT uses an artificial neural network that trains on data sets from billions of webpages from the internet, with trillions of words, and selects a word that statistically will follow its predecessor. Artificial Neural Networks have been around for a long time, and they are excellent at pattern matching. In the case of ChatGPT, the program weights all the words that have been connected to the input and selects the word that has the highest probability the weights have assigned to it. The program then looks for the next word the same way.

GPT-3 (the program the runs ChatGPT) used over 3000 HGX A100 servers with over 28,000 GPS to train on over 570 GB of text data to train and assign weights to words. With months of training, it created over 170 billion connections between all the words and added weights representing the importance of each connected relative to the word. This level of computing was very costly, at about $500,000/day and using more electricity than 150,000 people use in a month (23M KWh). The trained Artificial Neural Network then had researchers ‘tweak’ the weights to help rule out odd responses and was trained until the responses were consistent to what the programmers want.

This results in a program that looks for the next -token prediction from a list of rated words as the word that will likely appear next. There is no semantic grounding, set knowledge base of information based on confirmed scientific data, or any criterion of truth other than that of the statistically probability of another work following your prompt given its large training set. For example, when you use CHATGPT you:

1. Give it a prompt
2. ChatGPT looks at the last word of the prompt and assigns a number to encode it.
3. It then multiplies that number by the connections of everything it learned how words are associated with each other (embedded) which creates a 12,000-dimensional matrix
4. Attention transformers that identify which words in the prompt should have more attention that others (such as nouns over adverbs)
5. “normalize” output to make it seem more like a matrix again
6. Feed results forward to another layer of attention transformers (repeat 95 times)
7. Produce one word
8. ChatGPT repeats steps 2-7 for the rest of the words in the prompt.

At no point does ChatGPT know what the question is.

There is no knowledge-base being consulted.


It just works assign weights to nodes in a matrix representing the probability of a word that is most likely to follow its predecessor and selecting the one with the highest value. These weights are assigned through a training algorithm, that includes a ‘randomness” to ensure that the text appears “fresh”, working with a training set, as well as researchers “tweak” the weights to help create desired results.

It seems clear that one of the desired results is to create a program that appears intelligent regardless of knowledge or being accurate. This might not be too surprising when we remember that ChatGPT is a product of a private company where perception of a successful product is key for profit and increasing stock prices. Moreover, there is a lot of pressure to sell these systems. The current costs of these AI systems are creating a money pit for investors while the product seems to be a solution looking for a problem

What Could Go Wrong?

Misinformation and the Lack of Truth

ChatGPT may be designed to present its information in a convincing way. Since the first chatbot, ELIZA, people tend to anthropomorphize and place trust in these software applications. This can be dangerous with a program that can produce misleading statements. While there is a disclaimer to ChatGPT, OPEN AI benefited from the hype while not warning of issues with their product.

ChatGPT is capable of generating a considerable amount of nonsense, such as:

“Some experts believe that the act of eating a sock helps the brain to come out of its altered state as a result of meditation.”

While this is noticeably absurd, other statements may not be obvious. ChatGPT has already been documented to fabricate information and to adamantly defend these fabrications. Often these cases are referred to as “hallucinations”, as the chatbot produces responses as though they are correct. Apparently ChatGPT has been so convincing, that already a lawyer has been caught citing cases hallucinated by the program. This was a career ending mistake.

OPENAI recommends that users check the output of ChatGPT. However, not producing accurate information seems to be a serious flaw of the tool. Instead, ChatGPT is producing vast amounts of text with a variable level of “truthiness”. Another term for its output would be fiction or misinformation. Others argue that its output conforms with the technical term coined by Harry Frankford – the term ‘bullshit ’. It is named this in reference to the game where actors deliberately try to convince others of a statement with no regard to its truth.

In small doses misinformation on the web is not a problem, however AI has already produced more text than humans have since the Guttenberg printing press. Each day ChatGPT produces approximately 4.5 billion words a day. This flood of information will make it harder to find accurate and truthful information on the web. This is harmful on many levels, including contributing to undermining democracies. If colleges are going to promote the whole scale adoption of this technology, we do have to consider the increase in misinformation and resources that this will produce. It may be wise to have a more tempered approach.

Whenever a new language model like ChatGPT comes out, it gets a lot of hype. However, how should we proceed? We are not faced with the dilemma of promoting the misinformation spread by encouraging it or banning all AI reminiscent of human history in Herbert’s book Dune. Another option includes employing a measured approach where we carefully employ AI to illustrate its flaws and prepare students for the future. This may include how to combat a vast amount of misinformation and should certainly include highlighting the importance of information literacy and research librarians, who are regularly under-utilized by students.

ChatGPT may be designed to present its information in a convincing way. Since the first chatbot, ELIZA could encourage people to anthropomorphize and place trust in the software application. This can be dangerous when a program that can produce misleading statements. While there is a disclaimer to ChatGPT, OPEN AI benefited from the hype while not warning of issues with their product.

ChatGPT is capable of generating a considerable amount of nonsense, such as:

“Some experts believe that the act of eating a sock helps the brain to come out of its altered state as a result of meditation.”

While this is noticeably absurd, other statements may not be obvious. ChatGPT has already been documented to fabricate information and to adamantly defend these fabrications. Often these cases are referred to as “hallucinations”, as the chatbot produces responses as though they are correct. Apparently ChatGPT has been so convincing, that already a lawyer has been caught citing cases hallucinated by the program. This was a career ending mistake.

OPENAI recommends that users check the output of ChatGPT.  However, not producing accurate information seems to be a serious flaw of the tool. Instead, ChatGPT is producing vast amounts of text with a variable level of “truthiness”.  Another term for its output would be fiction or misinformation. Others argue that its output conforms with the technical term coined by Harry Frankford – it is  ‘bullshit’.  

In small doses misinformation on the web is not a problem, however AI has already produced more text than humans have since the Guttenberg printing press. Each day ChatGPT produces approximately 4.5 billion words a day. This flood of information will make it harder to find accurate and truthful information on the web. This is harmful on many levels, including contributing to undermining democracies. If colleges are going to promote the whole scale adoption of this technology, we do have to consider the increase in misinformation and resources that this will produce. It may be wise to have a more tempered approach.

Whenever a new language model like ChatGPT comes out, it gets a lot of hype. However, how should we proceed? We are not faced with the dilemma of promoting the misinformation spread by encouraging it or banning all AI reminiscent of human history in Herbert’ book Dune. Another option includes employing a measured approach where we carefully employ AI to illustrate its flaws and prepare students for the future. This may include how to combat a vast amount of misinformation and should certainly include highlighting the importance of information literacy and research librarians, who are regularly under-utilized by students.

Reference  

Berry, D (2018) "Weizenbaum, ELIZA and the End of Human Reason". In Baranovska, Marianna; Höltgen, Stefan (eds.). Hello, I'm Eliza: Fünfzig Jahre Gespräche mit Computern [Hello, I'm Eliza: Fifty Years of Conversations with Computers] (in German) (1st ed.). Berlin: Projekt Verlag. pp. 53–70.

Bobrowsky, M (2023) The Metaverse is Quickly Turning into the Meh-taverse , The Wall Street Journal. Mar 29.

Colom R, Karama S, Jung RE, Haier RJ. (2010) Human intelligence and brain networks. Dialogues Clin Neurosci. 12(4):489-501

DeGeurin, M (2023) 'Thirsty' AI: Training ChatGPT Required Enough Water to Fill a Nuclear Reactor's Cooling Tower, Study Finds. Gizmodo. May 10.

Dotan, T (2023) Big Tech Struggles to Turn AI Hype Into Profits. The Wall Street Journal. Oct 9.

Ellingrud, K., Sanghvi, S. Dandona, G, Madgavkar, A, Chu, M, White, O and P. Hasebe (2023) Generative AI and the Future of Work in America. McKinsey Global Institute. July 25.

Estreich, G (2019) Fables and Futures: Biotechnology, Disability, and the Stories We Tell Ourselves, Cambridge, MA: MIT Press.

Frankfurt, H. G. (1988) “On Bullshit.” The Importance of What We Care About: Philosophical Essays, pp. 117–133. Cambridge: Cambridge University Press (originally published in the Raritan Quarterly Review, 6(2): 81–100, 1986; reprinted as a book in 2005 by Princeton University Press).

Gebru, T, Morgenstern J, Vecchione, B, Wortman Vaughan, J, Wallack, H, Daume, H, and K. Crawford (2022) Excerpt from Datasheets for Datasets. In Ethics of Data and Analytics: Concepts and Cases. Martin, K. New York: Auerbach Publications

Hicks, M (2023) No, ChatGPT Can’t be Your New Research Assistant. Chronicle of Higher Education. Ag 23.

Hetzner, C (2023) ChatGPT moves to cash in on its fame as OpenAI launches plan to charge monthly fee for premium subscribers. Fortune. January 23.


Hopfield, J. J. (1982). "Neural networks and physical systems with emergent collective computational abilities". Proceedings of the National Academy of Sciences. 79 (8): 2554–2558

Knight, W (2022) ChatGPT’s Most Charming Trick Is Also Its Biggest Flaw. Wired. Dec 7.

Korhonen, P (2023) AI is a solution in search of a problem. UXDesign. Feb 2

Ludvigsen (2023) ChatGPT’s Electricity Consumption, Part II Medium. Mar 5

Magesh, V (2024) AI on Trial: Legal Models Hallucinate in 1 out of 6 (or More) Benchmarking Queries, HAI. Stanford University. May 23.

McDermott, D (1976) Artificial Intelligence Meets Natural Stupidity. SIGART Newsletter, 57:4-9

McMurtrie, B (2023) Teaching: Want your students to be skeptical of ChatGPT? Chronical of Higher Education. Sep 21

Merken, S (2023) Lawyer who cited cases concocted by AI asks judge to spare sanctions. Reuters. June 8.

Mollick, E (2023) Centaurs and Cyborgs on the Jagged Frontier. One Useful Thing, Sep 16.

Morgan, S. (2018) Fake news, disinformation, manipulation and online tactics to undermine democracy, Journal of Cyber Policy, 3:1, 39-43.

Newell, A. (1963). A Guide to the General Problem-Solver Program GPS-2-2. RAND Corporation, Santa Monica, California. Technical Report No. RM-3337-PR.

Newell, A.; Shaw, J.C.; Simon, H.A. (1959). Report on a general problem-solving program. Proceedings of the International Conference on Information Processing. pp. 256–264.

Norvig, Peter (1992), ELIZA: Paradigms of Artificial Intelligence Programming, San Francisco: Morgan Kaufmann Publishers

Olson, P. (2023) There’s No Such Thing as Artificial Intelligence. The Washington Post. Mar 26.

Ouyang, L, et al (2022) Training language models to follow instructions with human feedback. OpenAI

Pawar, S (2023) ChatGPT Costs a Whopping $700,000/Day to Operate, Says Research Analytics Drift. April 25.

Roose, K. (2023) Don’t ban ChatGPT in Schools. Teach with it. The New York Times. Jan. 12

Ropek, L (2023) So Far, AI Is a Money Pit That Isn't Paying Off Gizmodo. Oct 10.

Vincent, J (2021) OpenAI’s text-generating system GPT-3 is now spewing out 4.5 billion words a day. The Verge. Mar 29.

Valdez, E (2023) How much is invested in Artificial Intelligence Exoinsight. Mar 23.

Wolfram, S. (2023) What is ChatGPT Doing … and Why Does It Work? Stephen Wolfram Writings.

Cheating the Benefits of Cheating with AI

With the popularity and extraordinary, and dubious, claims of the ability of AI (Narayanan & Kapoor, 2024), a heightened concern about students cheating has occurred. Naturally, proctoring and anti-plagiarism companies benefit from this fear and the sales of their ‘solutions’ increase.  The problem with this policing solution is that beyond a point, teachers get into an ‘arms race’ between AI and AI-detection software where the real loser is the institution footing the bill.  One might think that the companies are happy to ‘stir the pot’ to increase anxiety for the benefit of their shareholders. Meanwhile, research has indicated that the level of academic dishonestly has not changed with the prevalence of AI (Lee, et al, 2024) Yet the focus on student policing places instructors in an adversarial relationship with the students, instead of an instructive one. 

 

A Better Solution

 

Low-stakes assignments have the added benefit of deterring academic dishonesty.  Because there is less risk involved in the assignment, there is less incentive to cheat, which could bring about sever consequences.  Why take the risk, when there is so little to gain?

 

Low-stakes assignments, such as threaded assignments, thwart using AI that quickly generates content.  The low-stakes assignments act as scaffold that requires further reflection and meta-cognitive skills that are not easily replicated by the stochastic language models modern AI uses. This will force the would-be culprit to reflect and expend so much energy producing something to submit that it isn’t worth the effort to get whatever the AI model can produce.

 

When assignments:

•  offer less stress, 
• supply tools for the students to succeed, 
• clearly express expectations, and 
• encourage the student to take control of their learning, 

the students begin to see the value in what they are learning.  This dissuades the learner from cutting corners instead of easily going through the process.  Not only will this lower academic dishonesty in your classes, low stakes assignments will encourage more of your students to become engaged active learners.

 

Low-stakes assignments engage the students in the learning process.  The best way to eliminate academic dishonestly is to remove the incentive to cheat and be up-front about the rules.  Students are often ‘cheating’ because often they have not received adequate instruction and expectations (Waltzer, Bareket-Shavit & Dahl, 2023).  To solve this, explicitly state the acceptable level of AI usage and you will curb the level of unintended violations to your academic expectations. A group of scaffolded low stakes assignments, often scaffolded to create a large assignment, then undermines the pay-off from cheating.

References

Futterman, K (2024) Zeitgeist 6.0: Results of the Campus sixth-annual student-body survey. The Middlebury Campus. Dec 13.

Lee, V., Pope, D., Miles, S, and R. Zarate (2024) Cheating in the age of generative AI: A high school survey study of cheating behaviors before and after the release of ChatGPT. Computers and Education: Artificial Intelligence. Vol 7, December.

Losey, R. (2024) Stochastic Parrots: How to tell if something was written by an AI or a human? 

Mintz, S (2023) 10 Ways to Prevent Cheating. Inside Higher Ed. February 16.

Mollick, E (2023) Centaurs and Cyborgs on the Jagged Frontier: I think we have an answer on whether AIs will reshape work. One Useful Thing. Sep. 16.

Narayanan, A & S. Kapoor (2024) AI Snake Oil: What artificial Intelligence can do, what it can’t, and how to tell the difference. Princeton University Press: Princeton, NJ.

Waltzer, T., Bareket-Shavit, C., & Dahl, A. (2023). Teaching the What, Why, and How of Academic Integrity: Naturalistic Evidence from College Classrooms. Journal of College and Character, 24(3), 261–284.

Wehlburg, K (2021) Assessment design that supports authentic learning (and discourages cheating) Times Higher Education. Nov 24