Optimizing operations involved in cell architecture

Abstract

Teams use inbuilt-monitors to cut tickets whenever their service metrics are breaching thresholds. There are separate alarms for each service metric for each cell. Whenever there is a regional issue (like an AWS LSE (Large Scale Event)), teams get a lot of tickets for their cell services having the same root-cause. Whenever there is a regional issue (like an AWS LSE (Large Scale Events)), teams get a multiple of tickets for their services with the same root-cause. This causes two problems: 1. During LSEs, it can lead to a lot of noise in the ticket queue and make focusing difficult for the on call. 2. All these tickets have to be closed manually. It is also possible to miss some tickets. We need to reduce the number of tickets corresponding to the same issue. As the number of silos per region increases, this will become more important. De-duping solutions should ideally have all these properties 1. We should reduce ticket count as much as possible 2. We should have one root cause per ticket 3. Implementation should be purely in terms of Carnaval (we’ll avoid introducing DJS jobs/services/etc. to avoid new points of failure)