![]() ![]() At a high level, we wanted ensure that we could monitor the different components of the application, understand performance parameters, and get alerted when things go wrong. We decided to host the Spark cluster using the Amazon EMR service, which manages a fleet of EC2 instances to run our data-processing pipelines.Īs part of preparing the cluster and application for deployment to production, we needed to implement monitoring so we could track the streaming application and the Spark infrastructure itself. The data consumed from Kafka comprises different types of telemetry events generated by mobile devices. ![]() ![]() We recently implemented a Spark streaming application, which consumes data from from multiple Kafka topics. Except when referring to specific metric names for clarity, we will replace this words with “primary”. When possible, Datadog does not use these terms. This post originally appeared on her blog.Įditor’s note: Apache uses the terms “master” to describe its architecture and certain metrics. This is a guest post by Priya Matpadi, Principal Engineer at Lookout, a mobile-first security platform for protecting mobile endpoints, consumer-facing apps, and more. ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |