WebDriver/RemoteProtocol/Meetings/2019/11/15

From MozillaWiki
Jump to: navigation, search

Agenda

  • Actions
    • Find out if there’s a way to enable logging of system observer notifications (ato)
    • Collect list of methods failing with UnknownMethodError from a recent Puppeteer test run (maja_zf)
  • Tracking MVP issues in GitHub on external repositories. (mmucci/maja_zf)
  • Project Schedule [mmucci]
    • Planning session will be held during the team meeting on Friday November 22.
    • Team will select the MVP Backlog based on a review of the team completion speed and forecast schedule.
  • Firefox Headlines [henrik]
  • Gutenberg
    • [dburns] Running tests to see where we are
      • We need to prioritise Bug 1595177 as that will unlock `links.test.js` file

Roster

Present
mmucci, whimboo, maja_zf, AutomatedTester, ato
Regrets

Minutes

Actions

ato
We had a couple of action items. The first one was on me from two weeks ago to figure out how to enable logging full-system observer notifications and I just landed the patch for this and it's not in the docks yet (see https://hg.mozilla.org/integration/autoland/rev/152910bc090b)
The second action was on Maja to collect a list of failing methods in Puppeteer unit tests.
maja_zf
It wasn't a very long list: Network.deleteCookies, Network.getCookies, Page.setInterceptFileChooserDialog
When you skimmed things, had you noticed something else?
ato
<no>
ato
I think that the Page.setInterceptFileChooserDialog is a low hanging fruit. Next on my to do list to work on setting files so I might look at it if I get some time.
maja_zf
Cool. Well, that that's that's my update on that. I also have been running the gutenberg tests. Some are still running in headless mode right now, they've been running overnight.
whimboo
I assume it's only because of the amount of timeouts we have now, right?
maja_zf
Yeah, so I set a shorter timeout. I haven't checked in detail, but it seems like better things are happening in headless mode than non-headless mode. Th non-headless mode tests failed spectacularly and finished much sooner so that's interesting.
ato
I wonder if that's because we're missing some of those preferences, because when you're running headless mode it like mocks out the windows and tabs, which makes everything much more reliable in terms of focus and different automation aspects.
maja_zf
I'm running Puppeteer with a bunch of preferences that I grabbed from Marionette, geckodriver and puppeteer-firefox, so that should not be it. I could still be missing some very important preferences, but that should not be the problem.
Here's a link that shows which prefs are being used in the gutenberg test run: https://hg.mozilla.org/try/rev/860cece0b54ed7a7364775c22e68933e0b796822
ato
This look this list looks quite extensive, so that means it's probably not preferences as you said.
maja_zf
Yeah, so I think there's a lot to investigate in the Gutenberg tests and if we can unblock a couple of the main failures, hopefully, we could make a big improvement and get some more meaningful results.
ato
Do you have a sense of what the main failures are?
AutomatedTester
So the one that I found today is Page.type; it doesn't appear to be working in Firefox. We're not getting through the login screen on one test file and there's like 20 tests in there. And so all those tests are timing out and for me that's where thing were taking so long yesterday when running in headless mode.
I've put the details in the agenda for this meeting
maja_zf
Page.type is Puppeteer method.
ato
Oh, I see.
maja_zf
So in if you if do Page.type with a special character, it call the Input.insertText CDP method, but that's not the case here. I believe that's only used in a couple of the tests and otherwise, it's Input.dispatchKeyEvent.
AutomatedTester
Yeah. In this case it wasn't even typing the word "admin", like that wasn't even going into the textbox. I haven't gone any further, but like
maja_zf
It must actually be a focus issue.
AutomatedTester
Or might be a focus. I don't know. I need to now rerun everything and try to watch which commands are going back and forth.
maja_zf
Yeah, I think a good thing to do would be to just focus on one sub test.
AutomatedTester
Yeah that's that's what I'm trying to do. I'm trying to just see a test actually happen and it's not happening. I can't even run that sub test because I can't get logged in.
maja_zf
The other big error is a Protocol error hit when calling connect in Launcher.js, see https://bugzilla.mozilla.org/show_bug.cgi?id=1596734
Sounds like they're connecting to a running browser instance, that's when launcher Connect is called. And it's with something with Target.setDiscoverTargets, so Henrik said this might be because of multiple tabs being opened because of missing preferences, but I don't think that's the case. But anyway, that's another thing to dig into
ato
So it sounds like there are a couple of known avenues that require more investigation. It's not as simple as just missing a command; it's more complicated than that.
maja_zf
Yeah, and the gutenberg tests are written in a way that's very... tricky to work with.

<general agreement>

Next Steps

ato
Right let's talk a little bit about about what's next. The two new methods aren't a particularly fruitful avenue for what is blocking Gutenberg next but but I'm going to work on at least one of one of them to do with file support
What is next on your list, Maja?
maja_zf
I'm going finish the preferences in the Puppeteer profile. I have a work in progress for setting up Firefox in the Puppeteer repo Travis setup and since I'm knee deep in Gutenberg, I think I should investigate these failures in more detail and file proper bugs and prioritize based on that.
ato
And I saw that Mathias also came back with some feedback on your PR
maja_zf
There's like little cosmetic things so I haven't responded yet, but I will try to finish that today as well.

Tracking MVP issues on external repositories

maja_zf
Yeah, so all the stuff I'm working on isn't tracked because it's not in repositories that we own. I have no way of indicating that I'm working on things.
ato
There's the option to file a bug to correspond with that work because we've done that for the security review and that's Marco's spreadsheet as a confidential bug. That bug is just meta bug, no patch will actually end up landing on that bug, but it will eventually get fixed.
maja_zf
Yeah, I'm fine with any approache. It's just a question of agreeing on how do we do this because we had previously talked about avoiding double accounting, but I don't think there's a way to avoid some double accounting in this case.
mmucci
I'm fine with a Bugzilla counterpart being followed. And I can track the bug and then when you start working on the GitHub issue just assign yourself to the bug, when you close the GitHub issue you resolve the bug.
I'm also fine if you just, you can just tell me the GitHub issue and I can put it on the dashboard, like we have for them right now on rows 32 to 35. I'm good with either. I can work with either way, the only thing is if we just go the GitHub way then= whoever's working on the GitHub issue should just to assign themselves when they actually start working on it so I can get the date timestamp from GitHub, then when it's done, just to close the issue as normal.
maja_zf
So this is in a repository, that's not owned by us, we're not collaborators, so I can't do that.
mmucci
Okay then. I, I would say, let's just let the easiest thing would be to file a corresponding Bugzilla bug.
maja_zf
That's fine with me. If the rest of the team is good.
whimboo
Another thing you should consider is, if your PR get merged on the GitHub repo and we would have to wait until this feature will be available for us. So should we close the bug immediately or wait until we have this fix in our tree?
maja_zf
Or maybe that could be part of the process that I'm the bug on Bugzilla also tracks updating the vendored Puppeteer code. We're not tied to any Puppeteer release.
ato
Yeah, the process is basically just to rebase from their master. As we discussed in the past, I feel perfectly fine with someone just updating the andreastt/Puppeteer firefox branch at any point and pushing experimental fixes, etc. to that branch. If you need review, that's fine.
ato
Related to this, I tried self-reviewing a documentation change amd the process for self review on Phabrictor now that we don't have inbound is quite confusing. I sent an email to dev.platform, so if you're curious about how self review works you should read that the email: https://groups.google.com/d/msg/mozilla.dev.platform/7_FUhNyI_4c/-jUycD8iBwAJ.
mmucci
So just to confirm the four GitHub issues on the dashboard now be replaced with four Bugzilla bugs. I'll just wait for you to file them, and then I'll swap them out.

Project Schedule

mmucci
We're all good for next week to review it. I'd like to make the first item "the team reviews". I'll have the full dashboard up and running and then I'll present to you what the completion rate of the team is. I usually present the team with one or several scenarios and then at that point, the team is discussing,"do we want to go further? a closer date?" Then, I'll give you the impact of how many bugs we should be selecting. Right now we're looking at a completion rate of three work days to complete one bug right now for the team.
If you were doing this today, then for example if we want the initial alpha release to be to December 20, I would tell you right there how many bugs we should aim for as the initial commitment for that MVP, but if you wanted it further, I can definitely calculate that.
We'll be doing all that in them meeting and then "separating the MVP from the reserve" discussions, and if you want to add contingency into the schedule. Contingency is basically a block number that would allow us to account for unknown bugs that we may be adding in the future. Say, if you had 20 known bugs listed as your MVP and 10 contingency, our MVP scope for that initial delivery would be 30. We just don't know what those 10 are right now. But as you add a bug to the MVP, that 10 becomes 9 and then nine becomes eight etc until we get to zero. And then we're full. So that gives you the freedom to say "we know there's going to be more bugs. We don't know what they are specifically, so we can add them freely without negatively impacting the schedule."
Most teams have a contingency, some don't plan to finish on a particular date with the the bugs that they know we want to release with and anything new that comes up, we'll just throw into the reserves and if there is something that is more important that comes up, we'll swap out work. So they'll pick the lowest priority MVP and swap it out for the new bug.
Our list that exists right now is is quite big, which is good. There's been a lot a lot of bugs added, 32 on there right now, so we're obviously not going to get all 32 done by the end of 2019
whimboo
We still have meta bugs tracked by the dashboard so it must be more like 58 bugs in total.
ato
Yeah, that demands for for a lot of work in and and just to expand them optimistic side.
That is basically work we haven't looked at the yet. The meta bugs could be expanded to many more bugs as we investigate. Another point is that the Remote Agent currently does not support Fission at all and I'm not sure how to track that.
Converting different subsystems to JS window actors is a large chunk of outstanding work that we haven't even began thinking about yet so we're not going to get that done by end of year. I think it's something that it makes sense to delay until start of 2020. But I just wanted to throw it in there, that there is like a huge chunk of work that we need to do at some point in order for the remote protocol to become useful for Real world test cases. At the moment, were targeting gutenberg which runs everything from the same domain, so no affected by this problem.
maja_zf
So we can expect to finish around 10 bugs before December 15th.
mmucci
Right, there will be 20 working days between next week and December 20th.
ato
We have to figure out our holidays as that will impact the projection if we lose one or two people in the last weeks.
whimboo
I will take off after 23rd.
maja_zf
Well, and also I don't think anyone wants to be rushing for a deadline on Friday, December 20. We should probably think about what we can do by December 16 and then the rest of that week is sort of damage control, so to speak.
ato
Yeah, I would hate for this to turn into a sprint. Seems like an artitrary thing to aim for. Rather, what we end up having on 20th of December is what we end up having, and then the work that we didn't have time to do will wait until the first week of January.
For myself, I don't have any plans for any holiday before the 20th, but I'm going to Norway over Christmas.
mmucci
What I what I recommend the teams regarding making a decision as to what is in the MVP, is you can think of it in two ways. Both ways work. You can focus on a date which then influences the amount of work will be that can be done up to that date, or you can focus on the amount of work that you want, which then determines the date.
Your team completion speed obviously gives you that final calculation in both scenarios. But if you are just looking for an end of your milestone, then there's going to be subsequent planning for the next milestone and and on, no problem.
If you're looking to that assert that the alpha deliverable has to be a certain amount of bugs for our list then it might be past December 20 and we'll just track it to completion.
So this is a discussion internally as a team, you need to have over the week.
David, if you were to just send me a message whether to give just the December 20 scenario or give us both scenarios, December 20 and past or if you actually knew a specific set of bugs that you wanted, I'll give you that scenario.
ato
It seems like this is a useful follow up to go into more detail about what actually we feel we need in order to release an MVP. Perhaps that's a good topic that we can we can do after your presentation next week.

Firefox Headlines Newsletter

whimboo
So as we discussed, we want to get our stuff added to the newsletter or the blog post.
It's for the public too and for this we have the Firefox meeting that's happening, each second week. We have to present some entries at the meeting, so I created a Google doc where we can add highlights in preparation for Firefox meeting and the only have to insert those lines into the wiki page which would make it easier for them, I think.
Maja, will get started with presenting. I tried to reach Harald to ask who should present, should it be a product manager, but haven't heard back. So Maja will present and get our stuff

added. The next meeting is next week Tuesday. Everyone, if I missed something feel free edit the Google Doc, write down bugs that have landed.

Other teams also report work from contributors. Also there is a separate section for the first-time contributions; you have to check their Bugzilla activity to verify this before listing contibutors there.
maja_zf
Um, let's just not add too much detail.
whimboo
This is the list of of bugs we have fixed and we can select the top ones <to present?>.
ato
I think, I think that the important thing we want to communicate is that some progress is happening on the remote protocol and we need to phrase that in a way that is understandable, so just pasting the bug titles might not be the right way to do it. You might have to do a bit of copy writing

so it makes sense to someone who doesn't know all the details.

maja_zf
Yeah, and for the first update I think I'll have to give a very short introduction to the project.
ato
I have at least like five or six different versions explaining what the remote control work is all about. I noticed the overview in our documentation is starting to become a little bit outdated. Whatever introduction we end up presenting at the Firefox meeting, it would make sense to try to consolidate that as much as possible and put it into the documentation as well so that we can reuse it.
maja_zf
Yeah, so as the person who's presenting this information at the meeting, my request is to please keep your additions to the doc relatively high-level and understandable to someone distant from the team. If you want me to emphasize something in the update, please indicate that.
ato
I think for the first meeting just giving an update about this team and what we're working on and what what we're trying to achieve would be enough. I mean you won't be able to do any updates.
whimboo
For sure, but in the written update we still need to include the stuff we have worked on in the past two weeks. If you don't have too much time for the presentation just introduct the project and the rest is just to read over read only.
maja_zf
Yes, but even for the read-only stuff, I don't think it's helpful to dump in everything, like it's supposed to be highlights for the rest of the community.
AutomatedTester
We could just make something bold when someone's adding it like this is important. And if everything is important, we only pick a random number. The first three items.
ato
This is a this is a good opportunity to make sure that our work gets known in the Firefox organization.

Performance

whimboo
I've seen me yesterday slow running of tests and issues with httpd.js. Connecting and disconnecting from the Remote Agent takes a very long time. I have seen this when I have cloned the chrome-remote-interface repository and ran their unit tests. They have nice output for commands that take a long time.
This is mainly for debug versions for sure, but even for opt builds it takes about 100 milliseconds. Through profiling a found it's a GC. I'm working with JavaScript people to figure out what we can do; xpcshell tests for media need GC between each of their tests.
It's interesting that all the tools you can use. You can figure out performance bottlenecks really easily. There's a even a profiler in devtools. There's a websocket viewer in devtools that should also help us figure out when something goes wrong.
ato
That's really great. Does it work with the websocket we have (there's no reason why it shouldn't) or only web documents? Have you tried it?
whimboo
Haven't tried it yet.

Actions

Status of Milestone 1

  • Last week: 51 Total; 45 Open (88.24%); 6 Resolved (11.76%); 0 Verified (0%)
  • This week: 58 Total; 49 Open (84.48%); 9 Resolved (15.52%); 0 Verified (0%)

Changelog

% git log --date=iso --pretty=format:'%ad%x09%H' -- remote/ | awk '$0 >= "2019-11-01" && $0 <= "2019-11-08"' | awk -F $'\t' '{print $2}' | xargs git show -s --format='%h%x09%an%x09%s'
23ec3d5f2cee    Ryan VanderMeulen       Bug 1594871 - Disable the racy sub-test. r=whimboo
121f19b7eb9b    Henrik Skupin   Bug 1587846 - [remote] Add "quality" option to Page.captureScreenshot. r=remote-protocol-reviewers,ato,maja_zf
b589179642b0    Henrik Skupin   Bug 1587846 - [remote] Add "format" option to Page.captureScreenshot. r=remote-protocol-reviewers,ato
d9081e2ef7da    Henrik Skupin   Bug 1587846 - [remote] Fix payload of return value for Page.captureScreenshot. r=remote-protocol-reviewers,ato
ed3629de9e8c    Henrik Skupin   Bug 1592643 - [remote] Implement Target.activateTarget. r=remote-protocol-reviewers,maja_zf,ato
c689ba0e46c8    Henrik Skupin   Bug 1591922 - [remote] Page.bringToFront has to wait for activate and focus events. r=remote-protocol-reviewers,ato
d06169c5aa69    Henrik Skupin   Bug 1592643 - [remote] Methods in Target domain have to raise for invalid "targetId" argument. r=remote-protocol-reviewers,ato
c50026801957    Henrik Skupin   Bug 1592643 - [remote] Refactor and improve browser chrome tests for Target domain. r=remote-protocol-reviewers,ato
15245002ac1f    Andreas Tolfsen bug 1549708: remote: implement Page.reload's ignoreCache argument; r=remote-protocol-reviewers,whimboo
a6683fe9a01c    Thomas  Bug 1591982 - Removed 'timestamp' property from Page.navigatedWithinDocument and Page.frameStoppedLoading r=whimboo,ato
5eef6dba8cc2    Andreas Tolfsen bug 1591927: remote: implement Security.setIgnoreCertificateErrors; r=remote-protocol-reviewers,maja_zf

Work

Milestones
Development status of Puppeteer alpha
Puppeteer alpha dashboard
Bugzilla queries
All project work currently in development
Available MVP work
Completed MVP work
Bug overviews
Gutenberg dependency tree
Puppeteer examples dependency tree
Complete Puppeteer dependency tree
All ze boogs

PTO (🍂)

Retrieved from "https://wiki.mozilla.org/index.php?title=WebDriver/RemoteProtocol/Meetings/2019/11/15&oldid=1234137"