Prodding ChatGPT into Writing Software
Edit: It's now March 2024. Many of my takes have not changed since GPT-4 and since when I wrote this. Chat GPT still takes wonky hallucinatory detours that make it mildly unreliable for doing stuff for you.
First: a quick note. Since I did this and as I finished writing this, GPT-4 for chatGPT shipped.
どうもありがとう 🤖
I've been using a piece of node.js stuff I found on Github to run a status webpage for my Valheim server for a year, give or take.
Last week, it stopped working. There was no obvious change in the logs that it read (permissions, format, whatever) nor did the script error out when launched.
The last commit was more than a year ago.
I had no desire to try to debug and fix the code, that being the case.
Let me also interrupt myself here to mention that I'm a Product Manager and a serial curious-to-a-fault-lets-try-this-out half-stuff-learner for anything avocational. Read: I can't code. Not in any meaningful way. I can grok code, I can tell you what code does if I look at it.
So, options that were on the table?
- Try and fix someone elses thing.
- Make my own thing from scratch
But: Why not have the robits do it?
Fig. 1: A Robit
I don't know if you heard, but ChatGPT and various other tools like Midjourney and elevenlabs.ai are all the rage right now. And also: all the controversy! Maybe sometime, but I don't reckon this is the space to get all verbose about the ethics of training sets for these things. Let it be known that I would never try to make a single Buck with software built in this way.
So, I found myself thinking that maybe this could be my way forward.
Last Friday, during my lunch break, I started cautiously.
First I asked it how it would approach the problem in the most general sense.
Okay, okay. Not too bad.
Then I asked it how I would expand the functionality a bit- sorting by date.
Interesting, okay. awk is a command that is powerful but I do overlook a lot. Nifty.
At this point I wanted to up the ante. I started treating the interaction less as a question-answer engine and more like a natural conversation. As if I was speaking to a flesh-and-blood developer.
ME: I have a log file in Linux. I want to write a script that will tell me if a player is logged in or not, based on that log file.
ME: I can tell if someone is logged in if :
A string that includes "Got connection SteamID" and an integer is not followed later on by a string that includes "Closing Socket" and the same integer.
I didn't go full-ham PM requirements guy on it, but I sprinkled in some more specification. I left out some things on purpose, like what language it should script in.
ChatGPT: To write a script that checks if a player is logged in based on a log file in Linux, you can use the grep command with regular expressions to search for the relevant strings and integers. Here's an example script:
#!/bin/bash
# Ask user for player ID
echo "Enter player ID (integer):"
read playerId
# Ask user for log file path
echo "Enter path to log file:"
read logFilePath
# Check for player login status
if grep -q "Got connection SteamID.*$playerId" "$logFilePath" && ! grep -q "Closing Socket.*$playerId" "$logFilePath"; then
echo "Player $playerId is logged in."
else
echo "Player $playerId is not logged in."
fi
# Explanation of the above command:
# - The first `grep` command uses a regular expression to search for a string that includes "Got connection SteamID" and the specified player ID integer.
# - The second `grep` command uses a regular expression to search for a string that includes "Closing Socket" and the same player ID integer.
# - The `!` operator in front of the second `grep` command negates the match, so the script checks for the absence of the closing socket string.
# - If the first `grep` command finds the player ID and the second `grep` command does not, the script displays a message indicating that the player is logged in. Otherwise, it displays a message indicating that the player is not logged in.
Save the above code in a file with a `.sh` extension (for example, `player_check.sh`) and make it executable.
Also not too shabby. Starting to get a little impressed, I asked it to make it generic for any playerID. That is to say- tell me if anybody is logged in.
#!/bin/bash
log_file="/home/steam/.config/unity3d/IronGate/Valheim/SKYROM/valheim_server.log"
steam_id=""
is_logged_in=false
while read line; do
if [[ $line == *"Got connection SteamID"* ]]; then
steam_id=$(echo $line | awk '{print $NF}')
is_logged_in=true
fi
if [[ $line == *"Closing socket"* && $line == *"$steam_id"* ]]; then
is_logged_in=false
fi
done < "$log_file"
if [ "$is_logged_in" = true ]; then
echo "A player is currently logged in."
else
echo "No player is currently logged in."
fi
That one took a little more poking and prodding. I was starting to discover a weakness that I will get to in a later section, which is the effect of the chaos or the fuzziness or the whatever-you-want-to-call-it leading to things like "change an entire function if you ask to just change part of it" and "unnecessary new functionality added for seemingly no reason"
More to the point, it was starting to feel like I was coming up against what I believe is a fundamental difference between a machine intelligence, even a good one, and a human mind. It can't reason. Decisions it makes are either based on clear logic, or they're just ultra arbitrary.
In this case, it had missed a hole in the logic, which is that the above wouldn't work properly if more than one person was logged in.
I pointed that out, and it immediately fixed it.
Knowing that the fundamentals worked, I proceeded to try and build a proper script and front-end combo. I did this bit by bit, with lots of back-and-forth with the chatbot instead of frontloading it with requirements.
Unfortunately, I deleted the threads because I ended up trying SEVEN different convos/threads before getting to something closer to what I wanted.
Again, because it would do things like change entire functions between responses without telling me. I had to start to pay closer attention to what it was doing.
But also, again, to its credit, when I pointed out one of those aberrant behviors (never remembering to set the encoding to the right one to read the logs) it (1) fixed it and (2) always brought it up with new code block generations.
This lead to a pretty funky flask app that it gave me.
from flask import Flask, render_template, jsonify
import json
import re
app = Flask(__name__)
log_file = '/home/steam/.config/unity3d/IronGate/Valheim/SKYROM/valheim_server.log'
players_file = 'players.json'
@app.route('/')
def index():
# Check if server is online
with open(log_file, 'r') as f:
log_content = f.read()
if 'Registering lobby' in log_content and 'Game - OnApplicationQuit' not in log_content:
server_status = 'ONLINE'
# Get online player IDs
player_id_regex = re.compile(r'Got connection SteamID (\d+)')
online_player_ids = set(player_id_regex.findall(log_content))
# Load player name map from JSON file
with open(players_file, 'r') as f:
players_map = json.load(f)
# Get online player names
online_player_names = []
for player_id in online_player_ids:
if player_id in players_map:
online_player_names.append(players_map[player_id])
# Get death count
death_regex = re.compile(r'Got character ZDOID from (\S+) : (\d+:\d+)')
death_count = len(death_regex.findall(log_content))
return render_template('index.html', server_status=server_status, online_players=online_player_names, death_count=death_count)
else:
server_status = 'OFFLINE'
return render_template('index.html', server_status=server_status)
if __name__ == '__main__':
app.run()
This was a little closer to what I needed/wanted. Had a separate HTML page that I could fiddle with, etc.
But after all this, I realized what I really needed to do was put the PM hat on.
I gave it a fairly detailed set of requirements.
It Worked
Actually first a quick note if you try this at home. I got rate limited by chatGPT and ended up shelling out 20 bucks to get the pro subscription. Worth it? YMMV. I don't remember how much it took before I got rate limited.
For your consideration, below is the full text of what I asked it to do, from the very start of a fresh chatGPT session.
I have an Ubuntu Linux server for a videogame called Valheim. It has a log file located at the path `/home/steam/.config/unity3d/IronGate/Valheim/SKYROM/valheim_server.log`.
I would like a single page website, using the Material Design Lite framework, that displays the following information.
The top of the page should indicate the server's status. This can be discerned from the following logic:
A log file line that includes the text "Game server connected" that is not later followed by a line that includes "OnApplicationQuit" means that the server is up. If "OnApplicationQuit" is present, the server is down.
The next item on the page should be a list of names of players that are currently online.
You can tell that a specific player is online if there is a log line "Got connection SteamID" followed by an integer, which is a unique player ID. As long as there is not a subsequent log line that includes "Closing socket" followed by the same player ID integer, it can be assumed that player is still online. If there are no players online, please display "No Vikings are Vike-ing".
The next item on the page should be a sum of player deaths since the server has been up, called "Skyrom Deadliness Index". In the log file, a player death looks like "Got character ZDOID from characterName : 0:0" where characterName is a string unique to the player that died.
The final item on the page should display the average save time for the server.
This can be discerned from the logs where the line looks like "World saved ( 1000.000ms )". The average can be for the time in that string in milliseconds for every time that line occurs in the log.
Please note that the log's timestamp format is specifically mm/dd/yyyy hh:mm:ss when parsing the log.
Please also note that I will have a JSON file available called players.json which maps player ID to a player name. Please tell me what the format should be.
I would like the front-end webpage to look modern, and poll for this information every ten seconds.
I might put together another post with the full back and forth from this point- but it's long. Very long.
With that said, from this point, it took a very minimal amount of fiddling to get to the state currently released as a v1.0 on my github account (https://github.com/Stillstellung/VikeAware).
Thoughts
ChatGPT:
- Doesn't seem able to "reason" as mentioned above. I think I'm okay with this.
- Intuition - it's there, but spotty.
- Competent at fixing errors- but not fantastic at noticing them in the first place
- It doesn't seem to really want you to make it write code. You have to "trick" it by saying things like "how would I do" "show me how" etc.
Things it's really good at or could be most helpful with:
- Natural language guide for using an unfamiliar API
- Regexes, queries, shell scripting, and the like
- Editing code for whitespace issues and missing characters or unclosed statements
All in all- did ChatGPT help me do anything I couldn't have done myself? Sort of. My JS is crap, so it was certainly a force multiplier and helped me build this limited thing pretty quickly, relatively speaking, once I knew how best to interact.
It was like pair programming with a smart but very literal person, over chat.
Less generously, like natural language comms with an embodiment of the dodgiest bits of stack exchange.
To that point, I think there's a lot of promise w/ things like GitHub Copilot. But do software engineers have a lot to fear? Maybe? But not quite yet. It feels like this thing is still only as good at the data set(s) it trained on.