Artificial Intelligence
Same Problem, Seven Years Apart: From Python Scripts to AI Agents

Introduction
In this post I describe two personal experiences, seven years apart, with the same problem: software that only lets you click, scroll, and type. I also compare two different approaches to solving this problem and discuss their implications. Before diving into the experiences, allow me to briefly explain what GUI automation is. If you are already familiar with the concept, feel free to skip the next paragraph.
GUI automation (Graphical User Interface automation) uses software to control and interact with graphical user interfaces automatically.1 GUI automation works by locating elements on the screen through (x, y) coordinates or screenshots, clicking buttons, navigating through pages, and typing in text. It's especially useful for repetitive tasks, automated testing and simulating workflows. Examples include automating data entry on web forms or simulating workflows of user stories after pushing a new software feature to a user acceptance testing (UAT) sandbox prior to a production deployment. Throughout my career, I've had opportunities present themselves to use GUI automation when no alternatives were available.
A Look into the Past
When I first began my tech career journey back in 2019, I was assigned the task of building mobile forms for our field technicians to conduct inspections on multifamily properties. The form builder was a graphical user interface (GUI) that provided a means for adding pages, sections, questions, conditional logic, and more. The field forms I created required granular logic to hide or make visible questions that were dependent upon a project's details such as program and pursuit. For example, if a user selected project A and it had program B plus pursuit C, those questions mapped to that specific combination of values needed to be visible to the inspector for that project while all other questions remained hidden. I had a spreadsheet full of thousands of questions and filter criteria and the list kept growing. Unfortunately, there was no way to programmatically create these questions or conditional logic, so you had to interact with the GUI directly by manually pointing-and-clicking and typing in the conditional logic values. As you can imagine, this was mind-numbing and time-consuming. During a team meeting, my former supervisor suggested I try to automate this task. Challenge accepted.
I began researching ways to automate this process. At this point, I already had foundational knowledge in programming, so I knew I'd write a script, but I didn't know which direction to take because there was no API or integrations to interact with. After hours of research, I stumbled upon GUI automation. Okay, now we're getting somewhere. Python also seemed like a good candidate for this task so I began searching for libraries that could take control of my screen and simulate what a human would do, which led me to discover PyAutoGUI.2 I had everything I needed: a computer, an integrated development environment (IDE), Python installed, my spreadsheet full of questions and filter criteria, the form builder, grit, and determination.
The algorithm was relatively simple. Through trial-and-error and a lot of testing, I developed an algorithm that extracted the values from the spreadsheet row-by-row and cell-by-cell then interacted with the web app GUI to point, click, and type in these values. I leveraged PyAutoGUI to use x-y coordinates and screenshots to find elements on the screen, type in values, and scroll to different areas, rinse and repeat. I met intermittent network issues and web app performance delays which would cause my script to break. I circumvented these issues by adding time delays between successive steps. This increased the amount of time it took for the automation to run but it increased the reliability and success rate from 50-60% to 80-90%. I would leave the script running overnight, come back in the morning, verify the results, manually correct anything that was missed, and improve the code iteratively after each run. Such a simple algorithm saved me so much headache and freed me up to focus on other important tasks.
Fast Forward to Today
Fast forward to today in the age of AI. I received a request to email out a project list for specific departments every two weeks. No problem, I thought. Our data already lived in our project management suite and this use case sounded like a suitable candidate for automation. With the capabilities of AI agents at my disposal, I began writing a prompt with details I thought the AI agent would need to build an automation of this nature. I gave it the context and off it went, sourcing information from our project management suite and asking me open-ended questions to fill in its knowledge gaps. The AI agent hit a blocker. It told me the connector didn't have the capability to create the automation. However, the AI agent offered to build the automation through its built-in web browser.
My cybersecurity training gave me pause, but my DevOps brain urged me to continue. I figured if I could monitor the AI agent every step of the way and give it least-privilege access to this low-risk software, I could mitigate potential security risks and minimize the associated impacts if it performed anything outside of the stated objective. It’s important to note the limitations of watching over an AI agent controlling your screen. Actions are being taken so quickly that it can be difficult to prevent them before they happen. This is why it’s so important to implement least-privilege access and test inside an isolated sandbox environment, when possible, especially if the system and software your AI agent is interacting with is inherently risky.
One click of approval and off the AI agent went. It took control of the browser, began screenshotting, and navigating to the proper page and elements on screen. It opened the automation, set the frequency and occurrence, built the query to source the project data, and generated the HTML for the email body. Although this wasn't a super complex automation to build, it was still an amazing use case, and it was neat to see an AI agent find an alternative solution through GUI automation when the typical approach was unavailable. Well done, robots. It created the automation exactly as I instructed it to. I just had to fill in the remaining information blanks that I intentionally left out to prevent accidental information exposure.
Comparing Different Approaches
During the build process and sometime after, I was mentally transported back seven years in the past when I wrote that script to automate the process of creating conditional logic in a form builder that only allowed interaction with the GUI. Times have certainly changed. Rather than writing a script, running it, and refactoring for improvements myself, the AI agent was able to navigate the GUI on its own, self-correct when it hit a blocker, and adapt to achieve its objective, all the while doing so in much less time than it would have taken me.
Writing standalone scripts is great when the GUI is repetitive and predictable. The downside is that once the layout of the GUI changes your script breaks and needs to be refactored. Using an AI agent for GUI automation is great for those one-off scenarios when no alternative is available and for scenarios where dynamic interaction is necessary. The downsides are that AI uses far more computing power than a standalone script, which increases costs. It also introduces potential security risks. The hybrid approach would be using AI to develop a GUI automation script, validating the script, triggering that script to interact with the GUI, and having AI assist in the refactoring process. Each of these approaches has merit and I find the application of each one to be entirely situational.
Conclusion
The power of this technology continues to amaze me, but it is also equally concerning. I can certainly understand people's apprehension towards its adoption, whatever their rationale is—from worrying about job displacement to cyber-attacks. No matter the application of this technology, it's important to assess where the human fits into the equation and how the human stays in the loop. It's my belief that this technology should be a supplement to our existing job duties so that we can be freed up to do the tasks that have the greatest impact on ourselves and our organizations. Despite its shortcomings, AI enables me to give more time towards the things that matter to me: continuous education, leadership, systems architecture design, security awareness, and others.
GUI automation is something you can fall back to when the front door is locked or there’s no door to begin with. GUI automation is just one side of the acceleration of AI capabilities, but it opens entirely new realms of possibilities when APIs and integrations cannot be reached. I'm excited to apply it to other use cases to free up not only my time but also others' time from those mindless, redundant tasks. What are your thoughts on GUI automation with and without AI involved?
