Now loading...
When product issues arise, customer support often serves as the initial indicator. Customers frequently report similar problems, sometimes even before engineering alerts are activated or formal incident declarations are made.
As the situation develops, the influx of support requests continues. Initially, support staff may not have the necessary information to provide customers. By the time a resolution is found, hundreds or even thousands of customers might be awaiting responses. It especially important for support teams to track these customers during the incident and ensure they receive accurate information once it is resolved.
The implementation of Grok Bot assists in managing these tasks efficiently. This AI support tool takes charge of operational tasks, specifically handling repetitive workflows that might otherwise fall between staff and systems. As previously noted in a recent blog, the SpaceXSI customer support team employs Support Bot within Grok Bot to aid in customer communication.
Recently, two additional AI components have been introduced for incident management. The first, IncidentWatch, surveils the support queue for surges in similar tickets and alerts the team when an anomaly is detected. The second, IncidentSweep, works on the opposite end, identifying affected tickets and processing them in approved batches.
During a notable incident on September 18, IncidentSweep was able to resolve 204 tickets in approximately ten minutes, with a support engineer overseeing and endorsing the batch. This process is vastly more efficient than the previous method, which entailed responding to tickets individually over many hours or using bulk responses that lacked precision.
Often, customers will detect problems ahead of internal alerts. A cluster of similar tickets can serve as an early warning of an incident. IncidentWatch, one of the recently developed AI assistants, is tasked with detecting these early signals. It evaluates ticket patterns every five minutes, analyzing the previous 15 minutes for any spikes related to the same issue. If a pattern is identified that does not match a known incident, IncidentWatch notifies the support engineer on duty, detailing the apparent issue, the number of related tickets, and examples for further investigation.
A support engineer then assesses whether these indications point to an actual incident and collaborates with the engineering team to explore the matter. During this period, IncidentWatch retains tickets that clearly correspond to the identified issue.
Once the incident is confirmed, it’s published on the public status page. Support Bot reverts to the held tickets, verifies them against the incident, and updates customers whose reports align. Throughout the incident, updates are communicated to keep customers informed.
After an incident is resolved, a high number of related tickets typically remain in the queue, each needing a final response. Traditionally, support teams could either reply individually—which could take hours—or use a bulk action, which, while quick, risks miscommunication by sending messages to unintended recipients.
IncidentSweep was designed to merge the advantages of both methods, allowing for both speed and accuracy. Support engineers can specify which tickets to review, which to leave out, and what message should be sent. After analyzing the tickets, IncidentSweep identifies those that are relevant and sets aside any that cannot be confidently associated with the incident.
Before dispatching any messages, IncidentSweep provides the engineer with a preview of the proposed batch, including the tickets selected and the message content. The engineer has the opportunity to verify and modify the list prior to final approval.
Once approved, IncidentSweep sends messages in manageable batches while documenting each action taken. It does not add any tickets post-approval, and in the event of a failure, it halts operations and logs the issue.
The Grok Bot framework offers a new approach to working with AI. Rather than merely relying on AI tools for singular tasks, teams can create AI counterparts that assume complete ownership of jobs and manage workflows through to completion.
This innovative model is particularly effective in customer support. IncidentWatch and IncidentSweep address tasks that once required manual oversight of a high number of individual tickets. SpaceXSI is actively exploring additional workflows to incorporate AI teammates in a similar capacity.
