Breakpad/WorkWeek

From MozillaWiki
Jump to: navigation, search

Mission

Make the web better.

  • Harvest crash report data
  • Generate aggregate statistics on data to understand trends
  • Publish individual reports for drill-downs and forensics
  • Determine the cause and frequency of all major crashes
  • Help Mozilla focus development resources in areas of highest impact

Goals this Week

  • Understand why reporter queries time out
  • Evaluate partitioning strategy
  • Evaluate hardware configuration
  • Evaluate backup and replication methods
  • Develop better/best practices for PostgreSQL

TODO

  • Performance
    • pg.conf changes
    • fs/os changes
    • query tuning
      • monitor
      • reports
      • interactive query tuning
        • memory checks
    • index/schema check
    • performance monitoring
      • install tools
      • check results
    • re-tuning & hw recommendations
    • application performance check
      • monitor/processor -- python
      • reporting -- php
    • full text for signatures
  • Failover
    • Uptime requirements
    • Prepare failover plan
    • Failover config & scripts
  • Partitioning
    • Partition scheme check
    • Check for bad queries
    • Rolloff plans
    • Deal with archive partition
  • IT Admin
    • Monitoring -- plans
      • Implementation
    • Backup & Redundancy plan
      • Implementation
  • Capacity Planning
    • (see above) Archiving Discussion
      • Implement archiving?
      • RRD?
    • Data growth analysis
    • New feature planning
  • DumpingDumpTables

Schedule

  • Tuesday
    • Partitioning
      • RRD/Archving?
    • Python Check
    • Full Text
    • Initial Replication
  • Wednesday
    • Continue replication
    • Query analysis etc.
    • Capacity planning data
    • Bug list and future feature planning
    • Interactive query checking
  • Thursday
    • Capacity Planning session
      • Schema changes?
    • PHP analysis?
    • Deploy replication?
  • Friday
    • Query analysis pt.2
    • Troubleshooting

Travel Schedules

  • aking
    • Arrive SJ @ 10:30 - will miss first half of Monday's meetings
    • Depart SJ @ 17:10 - cab @ 3pmish
    • Wild Palms
  • lars
    • Arrive SJ @ 17:00 Sunday, March 8
    • Depart SF @ 22:12 Saturday, March 14
    • Sweaty Palms
  • morgamic
    • Arrive SJ @ 22:30 Sunday, March 8
    • Depart SJ @ 19:20 Friday, March 13
    • Wild Palms

Indexes to Remove

  • productdims.productdims_product_version_key (not used, table is small)
  • reports_########.reports_########_product_version_key
  • topcrashers.idx_topcrashers_total
  • urldims.urldims_pkay
  • drop urldims.id altogether
  • urldims.urldims_url_domain_key
  • create pkey on url

Notable Queries

  • select * from pg_stat_user_tables where seq_scan > 100;
  • select * from pg_stat_user_tables where seq_scan > 100 and pg_relation_size(relname) > 800000;
  • select * from pg_statio_user_tables where relname = 'urldims';