Super Agent behavior bugs
I
Isaac Bruss
The Super Agent AI responses have some unintended buggy behavior. Here are three cases in order of increasing severity:
1) In the Super Agent Builder, it asked 2 questions to the user, and the suggested Follow Up responses attempted to answer both. But the 1st question isn't answerable by multiple-choice, so the follow up responses repeated the option "Use this list for tasks". Furthermore, I was baffled by what was meant by "this", because "this" wasn't defined, and I thought the agent was referencing something I couldn't see. See image, "clickup weird question.png". The fix here is a slight tweak to the prompt on how the AI should format questions to the user and how to handle follow up suggestions for questions that can't be answered by multiple-choice.
2) In Brain, the purpose of the 3 suggested follow ups in its system prompt got interpreted incorrectly. Here you can see that the 3 follow ups are 3 things the AI could ask the user, rather than 3 things the user could ask the AI. See image, "poor followup questions.png". The fix here is a tweak to the system prompt that clearly define the roles of the "user" and the "AI", and that the follow up suggestions are for the former and not the latter.
3) A potentially serious issue. I made a Super Agent that spawns tasks in batches. I then used that agent to create 500 tasks with a single prompt, and it worked! See image, "500 clickup tasks.png". I was going to test how high I could push this number, but I ran out of free credits. However, if I could do 500, then potentially there is no limit, and a curious user could use a Super Agent to spawn Billions of tasks and flood the system and/or burn all their credits with a single prompt. Even at <500 tasks I was getting cases of breaking the Tasks webpage and causing it to freeze, even after clicking the browser's refresh button. (I could restore it only by killing the Tasks page and re-navigating to it from my home page). The fix here would be a combination of light prompt optimization to ward against some of the obvious malicious prompt-hacking techniques, combined with real limits on agent tool use. You'll also have to provide the AI in its context an understanding of what those limits are, lest well-intentioned users hit these limits and neither they nor the agents understand why they are blocked.
Log In