Loop/Test/Monitoring
From MozillaWiki
Contents
Overview
The purpose of this wiki is to provide a reference of monitoring tools for both test and production environments. Typically monitoring would only be performed where a load is present.
This page will include some information about analysis (i.e. log analysis) as well.
Monitoring Tools (STAGE)
Loadtest (loadv1) Test Dashboard
Sentry:
- https://sentry.shared.us-east-1.stage.mozaws.net
- Getting access to Sentry STAGE - talk to the OPs team
Stackdriver:
Graphite
- https://graphite.shared.us-east-1.stage.mozaws.net
- Graphite > stats > loop-activated-users
- Graphite > stats > loop-call-urls
- User Graphs > not currently defined
Kibana
- https://kibana.shared.us-east-1.stage.mozaws.net/#/dashboard/file/landing.json
- https://kibana.shared.us-east-1.stage.mozaws.net/index.html#/dashboard/file/loop_http_status.json
Heka
- https://heka.shared.us-east-1.stage.mozaws.net/
- (or https://heka.shared.us-east-1.stage.mozaws.net/#health)
- PlugIns - Filters
- https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopBasicMetrics
- https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopHTTPStatus
- https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopActiveDailyCallers
- https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopTotalUsers
- https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopUrls
- https://heka.shared.us-east-1.stage.mozaws.net/#plugins/filters/LoopActiveDailySubscribers
- PlugIns - Outputs
- PlugIns - Encoders
- https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopNginxFileOutput-ProtobufEncoder
- https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopNginxErrorFileOutput-ProtobufEncoder
- https://heka.shared.us-east-1.stage.mozaws.net/#plugins/encoders/LoopServerFileOutput-ProtobufEncoder
- Sandboxes
- https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes
- https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_total_callers.json
- https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_cumulative_callers.json
- https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_daily_unique_callers.json
- https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_daily_unique_subscribers.json
- https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_new_callers.json
- https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_weekly_recurring.json
- https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_fortnightly_recurring.json
- https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_monthly_recurring.json
- https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_urls_clicked.json
- https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopBasicMetrics/outputs/LoopBasicMetrics.loop_urls_generated.json
- https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopHTTPStatus/outputs/LoopHTTPStatus.HTTPStatus.cbuf
- https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailyCallers/outputs/LoopActiveDailyCallers.EstimatedActiveDailyCallers.cbuf
- https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailyCallers/outputs/LoopActiveDailyCallers.EstimatedActiveDailyFxOSCallers.cbuf
- https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopTotalUsers/outputs/LoopTotalUsers.EstimatedTotalUsers.cbuf
- https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopTotalUsers/outputs/LoopTotalUsers.EstimatedTotalFxOSUsers.cbuf
- https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailySubscribers/outputs/LoopActiveDailySubscribers.EstimatedActiveDailySubscribers.cbuf
- https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopActiveDailySubscribers/outputs/LoopActiveDailySubscribers.EstimatedActiveDailyFxOSSubscribers.cbuf
- https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopUrls/outputs/LoopUrls.All.cbuf
- https://heka.shared.us-east-1.stage.mozaws.net/#sandboxes/LoopUrls/outputs/LoopUrls.FirefoxOS.cbuf
- PlugIns - Filters
Monitoring Tools (PRODUCTION)
Sentry
- http://sentry.shared.us-west-2.prod.mozaws.net
- Getting access to Sentry Prod - talk to the OPs team
- http://sentry.shared.us-west-2.prod.mozaws.net/loop-prod/
- http://sentry.shared.us-west-2.prod.mozaws.net/loop-prod/loop/
Stackdriver
Kibana
Heka
- Main: https://heka.shared.us-west-2.prod.mozaws.net/#
- https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/LoopActiveDailyUsers
- https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/LoopHTTPStatus
- https://heka.shared.us-west-2.prod.mozaws.net/#plugins/filters/MsisdnGatewayHTTPStatus
loop-server Dashboard
Graphite:
- https://graphite.shared.us-west-2.prod.mozaws.net (statsd data)
Firefox Input Dashboard:
Pingdom:
- loop-client: http://stats.pingdom.com/20dar76w4hmv/1304575
- loop-server: http://stats.pingdom.com/20dar76w4hmv/1304565
Log Monitoring
Tail multiple logs
$ cd /media/ephemeral0/nginx/logs $ tail -f loop_server.access.log loop_server.error.log default.access.log default.error.log
Other logs of interest:
For example, for loop-server:
- /var/log/circus.log
- /var/log/loop_err.log
- /var/log/loop_out.log
- /var/log/hekad/loop.stdout.log
- /var/log/hekad/loop.stderr.log
- /var/log/nginx/access.log
- /var/log/nginx/error.log
HTTP Access Log - Parsing
unique REST activity with counts
$ cat /media/ephemeral0/nginx/logs/loop_server.access.log | grep "HTTP/" | awk '{print $6" "$3" "}' | sort | uniq -c
unique REST activity with counts (from a .gz file)
$ zgrep -a "HTTP/" /media/ephemeral0/nginx/logs/loop_server.access.log-20150114.gz | awk '{print $6" "$3" "}' | sort | uniq -c
return only the interesting stuff
$ cat loop_server.access.log | grep -v '200 ' | grep -v '101 ' | grep -v '499 ' | grep -v '201 ' | grep -v '204 '