EngineeringProductivity/Projects/Perfherder

From MozillaWiki
Jump to: navigation, search

About

Perfherder is an interactive dashboard intended to allow monitoring and analysis of automated performance tests run against Mozilla products (currently Firefox and Firefox for Android).

Perfherder is part of the Treeherder project. The production version of it lives at https://treeherder.mozilla.org/perf.html

Team

  • Sheriffs (:igoldan, :bebe)
  • Developers (:sclements, :igoldan)

Goals

Develop tools and integration of them to assist sheriffs to find and report regressions to developers, and to assist developers in understanding the impact of their changes and potential fixes.

This includes:

  • Graphical display of performance data over time
  • An easy way to compare all performance tests for a given revision to another revision (or time range)
  • An alert system to provide notification when there has been a level shift in performance data, along with tools for filing bugs when such events occur
  • Integration with normal developer work flows for using the Try server and landing code

Non Goals

  • fixing tests
  • fixing harnesses
  • adding new tests
  • bisection of a range of commits to find the root cause

Dependencies / Who will use this

We have a dependency upon tools to generate data. Currently this is Talos, but the design is to allow all other types of tools to post here as well.

The other big dependency we have is TreeHerder. Right now we are tightly integrated into the project and use a lot of common data (branches, revisions, history, pushlog). We fully expect this to remain the case and will do our best to integrate as much as possible if it helps the workflow of the users.

The primary users will be:

  • sheriffs
  • developers

Secondary users will be:

  • manager (project, release)
  • qa

Contribution

We welcome contributions to Perfherder! We track issues inside the Perfherder component on bugzilla, see below for a list of open bugs.

The contribution section of the Treeherder page has some handy resources for those just getting started. In general, if you have a development version of treeherder running, you're all set to hack on Perfherder! Just jump to:

If you have a question, jump on #perfherder on irc.mozilla.org and look for :igoldan or :davehunt.

Source and docs

See the Treeherder page.

Bug Table

Full Query
ID Priority Component Summary Assigned to
1564833 P1 Perfherder [meta] Server side search in Alerts view Alexandru Irimovici
1830472 P1 Perfherder Pull time duration for the sheriffing KPIs alert categories Acasandrei Beatrice (needinfo me)
1747357 P1 Perfherder Allow storage for replicates Greg Mierzwinski [:sparky]
1852619 P2 Perfherder When copying summary for comment 0, percentages less than 1 do not have a link Andrej (:aglavic)
1523037 P2 Perfherder Page titles for subtests are confusing Acasandrei Beatrice (needinfo me)
1651432 P2 Perfherder Perfherder UI tests need to be updated Acasandrei Beatrice (needinfo me)
1694335 P2 Perfherder Delete the remove_vcs_data management command Acasandrei Beatrice (needinfo me)
1711883 P2 Perfherder Compare View call to backend responds with 502 Bad Gateway Acasandrei Beatrice (needinfo me)
1689621 P2 Perfherder [meta] Monitor backfill bot’ stability on production Ionuț Goldan [:igoldan]
1723719 P2 Perfherder Link to devtools test documentation Julian Descottes [:jdescottes]
1722092 P2 Perfherder comparison browsertime (Btime) metrics of Try push with mozilla-central fails to load data, Perfherder request of m-c data fails with server error
1824750 P2 Perfherder Add an API endpoint for getting the application version
1831943 P2 Perfherder Expand replicate storage to other projects/branches
1837970 P2 Perfherder Decrease try performance data expiration from 6 weeks to 3 weeks
1838025 P2 Perfherder Provide a method to filter out noisy/outlier data points in the compare view
1839150 P2 Perfherder Add side-by-side API endpoint
1848885 P2 Perfherder Add mention of ./mach try perf --alert to comment 0
1907909 P2 Perfherder Alert summaries with large number of alerting subtests show very large header
1932360 P2 Perfherder Remove the raptor framework
1931234 P2 Perfherder On perfherder, datapoints generated by backfill for AWSY tests do not have the "delta" number
1930819 P2 Perfherder Sometimes the link from AWFY-to-perfherder shows extra/incorrect/duplicate graphs for m-c
1930128 P2 Perfherder Perfherder ingestion error: duplicate key values for performance_alert_summary table insertion
1929962 P2 Perfherder Add replicates as an option/field to the suites in the performance artifact schema
1929249 P2 Perfherder Build a process for taking action on performance regressions that were resolved as WONTFIX/FIXED
1870682 P2 Perfherder Add links to docs for mozperftest tests in alerts view
1927867 P2 Perfherder Perfherder Graphs view fails to load properly when values include negative values
1919588 P2 Perfherder Add a new alert table for testing new detection methods
1915403 P2 Perfherder Add a small dashboard or redash query to keep up-to-date with alerts
1894252 P2 Perfherder AWSY test is showing subtest in the suite name with subtests under it as well
1915401 P2 Perfherder Modify wording in comment zero regarding backouts/plan
1914258 P2 Perfherder Add test owner/team in the cc for when their tests alert
1900606 P2 Perfherder Gather replicates during alert generation if they are available
1914167 P2 Perfherder Provide a link to compareview/perfcompare in the alert comment
1907925 P2 Perfherder Annotate profiles in before/after with before/after to denote which is which
1933686 P2 Perfherder Automatic status update not working properly
1905930 P2 Perfherder Filling alert bug should only use test names that regressed in bug title
1904551 P2 Perfherder Incorrect autogenerated summary on performance alert https://treeherder.mozilla.org/perfherder/alerts?id=853
1894726 P3 Perfherder Compare View should allow us to change the method used to summarize the values
1922516 P3 Perfherder Show whether a metric produces alerts in the graph view
1922517 P3 Perfherder Add documentation links for build_metrics alerts
1874894 P3 Perfherder Allow us to annotate data points in the graph view
1869245 P3 Perfherder Show common alerts along with the normal alert in graph tooltip
1867284 P3 Perfherder Add tooltip to "Use replicates" button
1855728 P3 Perfherder Add Slack and/or Element notifications for performance alerts
1265381 P3 Perfherder Consider alerting with less than 12 future data points if we're very confident there is a regression
1500528 P3 Perfherder Make perfherder comparisons consistent across benchmarks
1501239 P3 Perfherder Test validity dashboard
1501241 P3 Perfherder [meta] Make states more manageable
1515338 P3 Perfherder State machine diagram
1519289 P3 Perfherder Remove usage of signature_hash in Perfherder
1520726 P3 Perfherder Check fixes for perf regressions
1520731 P3 Perfherder Directly retrigger & backfill
1528740 P3 Perfherder Notify on new alerts/no more data from existing alerts
1532283 P3 Perfherder Improve & re enable nudge
1532995 P3 Perfherder Cover compare replicates view with FE tests
1539854 P3 Perfherder Timeranges in Perfherder URL are limited to 1, 2, 7, 14, 30, 60, 90 days
1543780 P3 Perfherder Make https://treeherder.mozilla.org/api/project/{repo}/performance/signatures faster
1546054 P3 Perfherder Automatically email top alerts on a weekly basis
1558746 P3 Perfherder Reflect confirmed commits on alerts
1562178 P3 Perfherder [meta] Automate backfills and retriggers for alerts
1565516 P3 Perfherder [meta] Improvements to compare view
1569018 P3 Perfherder Graphs of try vs try pushes show irrelevant comparison points
1569022 P3 Perfherder graph backdrop for pushes should be based on base revision, not push date
1569024 P3 Perfherder select the right test for the graph backdrop when adding more test data to graph
1569590 P3 Perfherder Mention pending retriggers in compare view
1573131 P3 Perfherder Automatically identify backouts
1581533 P3 Perfherder Discussion: Improving performance regression detection
1582711 P3 Perfherder [meta] Introduce noise alerts for perf tests
1585650 P3 Perfherder [meta] Provide boxplots in Graphs view
1587766 P3 Perfherder build times should be grouped by dimension sccache hit rate
1591045 P3 Perfherder Backfill reports should consider indirect alerts also
1591053 P3 Perfherder Change logic for the Graphs view "Add related configurations" hyperlinks
1593206 P3 Perfherder Make application fields mandatory in validation schema
1600668 P3 Perfherder Error out if PERFHERDER_DATA is malformed
1601952 P3 Perfherder Use The Mann-Whitney-U test
1605247 P3 Perfherder Consider adding more options for date ranges
1611203 P3 Perfherder Make it possible to embed Perfherder graphs in other pages
1615267 P3 Perfherder Tests View: Consistent renaming from health/validity to Tests View
1615268 P3 Perfherder [meta] Improvements to Tests View
1616191 P3 Perfherder Provide server side filtering by option
1616240 P3 Perfherder Provide backend test coverage for server side search
1616242 P3 Perfherder Provide frontend test coverage for server side search
1620736 P3 Perfherder Graphs View - Add Table view param in URL
1620737 P3 Perfherder Graphs View - When toggle to Table View, it should focus on table
1620739 P3 Perfherder Graphs View - TableView and GraphTooltip component share helpers functions
1620741 P3 Perfherder Graphs View - Table view could have trends and outliers navigation to improve accessibility
1622564 P3 Perfherder Improve performance of graphs view when showing a large date range
1622567 P3 Perfherder Add trend lines to graphs
1622568 P3 Perfherder Allow base/new revisions to be edited from the compare view
1622569 P3 Perfherder Add graphical plots for subtests
1622573 P3 Perfherder Data Review of Compare View
1622578 P3 Perfherder Add the ability to use the shortened hash for a commit in the compare view
1622829 P3 Perfherder Consider changing default display to more than 2 days
1623168 P3 Perfherder Expose information on modality of the results and alert when modality changes between results
1627922 P3 Perfherder Reassign Alerts dialog doesn't have autofocus
1633660 P3 Perfherder Attribute inconsistency in alert summary serializer
1635085 P3 Perfherder Provide test coverage for when Compare view is changing its URL
1635127 P3 Perfherder Alert summaries should show the status of linked bug
1639112 P3 Perfherder Consider making SecretaryTool a singleton
1639248 P3 Perfherder Feature request to save perfherder comparisons for future lookup
1639277 P3 Perfherder tp6 tests unnecessarily require multiple tasks to increase confidence on compare chooser
1639308 P3 Perfherder Show the last commit message when comparing two revisions
1641623 P3 Perfherder Add a numerical summary of Perfherder improvement/regression to the Compare Revisions view
1644707 P3 Perfherder Perfherder: Add test for platform dropdown options order
1646259 P3 Perfherder Add/modify perfherder graph view to better communicate noise
1649156 P3 Perfherder Distinguish subtest items from test items in alert summaries
1653624 P3 Perfherder add ability to change statistical method in use on perfherder compare view
1655019 P3 Perfherder Display a name/unit for summary results
1665100 P3 Perfherder Perherder should be able to show task name for a graph
1667888 P3 Perfherder Add test descriptions to Tests View in Perfherder
1684946 P3 Perfherder browsertime vismet tasks with --rebuild flag all reuse a single test result. Perfherder says Confidence is Infinity (high).
1689584 P3 Perfherder Provide POC for integrating Mann-Whitney-U
1689586 P3 Perfherder Provide actual plan for integrating MWU
1693051 P3 Perfherder Provide geomean extractor subtest in “free retriggers” suites
1693053 P3 Perfherder Integrate new strategy for displaying “free retriggers” from subtests
1700970 P3 Perfherder Perfherder compare Retrigger Jobs button no longer retriggers jobs
1701615 P3 Perfherder Group alert items in tests+subtests to visually distinguish them easier.
1723073 P3 Perfherder Have properties to distinguish manual/auto backfilled jobs
1724140 P3 Perfherder Show the graph with runs for before/after at the same time
1727787 P3 Perfherder Perfherder should show `unit` and `lower_is_better` in addition to existing color-coding on e.g. the Alert view
1728295 P3 Perfherder Improve test coverage for report_backfill_outcome command
1729665 P3 Perfherder Add a performance tab in Push Health
1734898 P3 Perfherder Add support for filtering out matches in perfherder compare
1741786 P3 Perfherder Performance regression detected between m-c build and try build with the same revision
1744024 P3 Perfherder fission/non-fission awsy jobs are mixed up in the display, and also in retrigger
1751158 P3 Perfherder Last page is not removed from pagination control when resolving a previous alert
1754151 P3 Perfherder Perfherder subtest in Compare View misses the first letter of the test name
1756774 P3 Perfherder Tests View - Changing the framework should reset the table to page 1
1773562 P3 Perfherder [perfdocs] Integrate perfdocs test owner information into our alerts
1837203 P3 Perfherder Add a way to tag infra changes
1838924 P3 Perfherder Integrate sitespeedio on treeherder for performance testing
1502805 P3 Perfherder Integrate browser notifications xuan pan
1901066 -- Perfherder When clicking a link from perfherder you should have the treeherder view filtered to just that job Andrej (:aglavic)
1902168 -- Perfherder Show reassigned alerts in graphs view
1905963 -- Perfherder Perfherder Compare Revisions "Show only important changes" criteria is different for improvements vs regressions
1901293 -- Perfherder Modified treeherder URL triggers an error with what looks like a typo: " Please try again in a minute.500:"
1901067 -- Perfherder Button to open all alerts on perfherder
1908641 -- Perfherder Perfherder's Tests View is currently broken
1912030 -- Perfherder Magnitude sorting on the alert summary page is confusing
1900303 -- Perfherder Two different values for the same alert metric in comment 0 when filling a sheriffing regression bug
1885014 -- Perfherder The "Use replicates" button on the graph view never works the first time you click it
1919432 -- Perfherder AWFY -Motionmark graphs are not updating since 16sep2024 (maybe because the old version of Motionmark was replaced by Motionmark1.3.1)
1877784 -- Perfherder Hovering over a replicate doesn't show you the value for that replicate
1924883 -- Perfherder Treeherder fails to deploy after latest database change: psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "django_content_type_pkey"
1860800 -- Perfherder Intermittent test_identify_retriggerables.py test_identify_retriggerables_selects_all_data_points assert {1, 2, 4, 7, 9} == {2, 4, 7, 9, 11}
1855574 -- Perfherder Error when clicking "View" for a talos-profiling task: `Something went wrong TypeError: can't access property "y", e[0] is undefined`
1931250 -- Perfherder Removing a graph which had an open popup throws an unrecoverable error "Something went wrong TypeError: can't access property "data", T is undefined"
1931377 -- Perfherder [GraphsView] shouldAlert field should be displayed in the Graph Legend
1854237 -- Perfherder Compare view: Typing ( into the filter text box causes the page to "crash"
1853410 -- Perfherder Reduce the radius of the plot points (graphs are too crowded)
1689373 -- Perfherder [meta] Provide plan for using Mann-Whitney-U alongside student' s t-test
1698545 -- Perfherder [meta] Link to test documentation
1709778 -- Perfherder [meta] Distinguish between auto/manual backfills
1807148 -- Perfherder Typo in Perfherder
1809467 -- Perfherder frequent intermittent failures on collapsable_rows_tests.jsx and test_perfcompare_api.py, disabled for now
1818857 -- Perfherder If retriggers fail on PerfHerder due to authentication issues, the error message gets thrown off the top of the viewport
1819632 -- Perfherder Hitting enter/return in test filter on perfherder graphs page shouldn't reload the page
1819634 -- Perfherder It's too difficult to look at m-c results for try graphs
1823531 -- Perfherder arewefastyet.com speedometer toplevel seems to be inverted
1831319 -- Perfherder Add percentage indicator to standard deviation in compare view
1836081 -- Perfherder perfherder graph does not show older data for damp signatures even if they are in the displayed timeframe
1839320 -- Perfherder Hide triaged alerts from untriaged queue
1842195 -- Perfherder Set status flag when filing regression bugs
1850748 -- Perfherder Add documentation about how to ingest performance data locally
1850756 -- Perfherder At certain window widths, the perfherder layout is broken
1850757 -- Perfherder Removing the first series leaves the UI in a broken state
1850759 -- Perfherder Rectangle drag selection gets stuck if the mouse is released outside the graph
1850764 -- Perfherder Pressing enter in the "filter tests" textbox should not close the panel

168 Total; 168 Open (100%); 0 Resolved (0%); 0 Verified (0%);

Retrieved from "https://wiki.mozilla.org/index.php?title=EngineeringProductivity/Projects/Perfherder&oldid=1213381"