AI Super Agents

Super Agent Not Ready for Primetime
I've been watching a ton of videos and community events to try and get a sense of actual use cases. In the end, most ended up being marketing hype more than anything OR "we use agents to do x, y, z", but never showing them in action. I finally decided to give one a shot last week. I started with a template (personal assistant I think). After some iteration on the setup, I thought I was set. Each morning it would send me my focus items for the day and a list of my meetings. Well it changed the format from the first day to the second and then again on the third (with no prompting). I decided I liked the second day the best and asked it to take what it gives in the chat and create a parent task with sub-tasks each day so I could actually check something off (the checklist in chat is not interactive). EVERY day since, it messes up something. The tasks gets created and it's in my Overdue section. Bot says "You're right...let me fix that" and says it'll be correct moving forward. Sometimes it's missing the subtasks, other times its missing tags, other times the meetings. When I call it out, it tells me I'm right and that it'll fix it and update it's preferences, but it continues to not be consistent and therefore not reliable. I've burned through 4K credit this week because the agent can't lock in and stay consistent with it's instructions. As the company's ClickUp "champion", I can't recommend this to anyone else on the team at this time. It's all great in theory but it's wasting more time and effort in fighting with the agent to get things right than is worth for the output it provides. Intrigued with the concept (I use AI for a lot of things), but disappointment with the experience so far. Lastly, I found it interesting when I had a call with some of the product team this week when I asked if/when do the credits renew, neither had an answer. I still don't know if they renew monthly or we have what we have and when we run out we have to purchase another batch of credits.
11
·
building now
Super Agent behavior bugs
The Super Agent AI responses have some unintended buggy behavior. Here are three cases in order of increasing severity: 1) In the Super Agent Builder, it asked 2 questions to the user, and the suggested Follow Up responses attempted to answer both. But the 1st question isn't answerable by multiple-choice, so the follow up responses repeated the option "Use this list for tasks". Furthermore, I was baffled by what was meant by "this", because "this" wasn't defined, and I thought the agent was referencing something I couldn't see. See image, "clickup weird question.png". The fix here is a slight tweak to the prompt on how the AI should format questions to the user and how to handle follow up suggestions for questions that can't be answered by multiple-choice. 2) In Brain, the purpose of the 3 suggested follow ups in its system prompt got interpreted incorrectly. Here you can see that the 3 follow ups are 3 things the AI could ask the user, rather than 3 things the user could ask the AI. See image, "poor followup questions.png". The fix here is a tweak to the system prompt that clearly define the roles of the "user" and the "AI", and that the follow up suggestions are for the former and not the latter. 3) A potentially serious issue. I made a Super Agent that spawns tasks in batches. I then used that agent to create 500 tasks with a single prompt, and it worked! See image, "500 clickup tasks.png". I was going to test how high I could push this number, but I ran out of free credits. However, if I could do 500, then potentially there is no limit, and a curious user could use a Super Agent to spawn Billions of tasks and flood the system and/or burn all their credits with a single prompt. Even at <500 tasks I was getting cases of breaking the Tasks webpage and causing it to freeze, even after clicking the browser's refresh button. (I could restore it only by killing the Tasks page and re-navigating to it from my home page). The fix here would be a combination of light prompt optimization to ward against some of the obvious malicious prompt-hacking techniques, combined with real limits on agent tool use. You'll also have to provide the AI in its context an understanding of what those limits are, lest well-intentioned users hit these limits and neither they nor the agents understand why they are blocked.
0
·
AI Super Agents
Load More