Paladin/Misc/Android Audio Notes

From MozillaWiki
Jump to: navigation, search

Here are some links from an IRC conversation about Android audio, and log of much of the conversation itself.

Log from channel # starting 2011-11-15 12:04:26 -0800

Link to a time in the conversation by visiting a URL like https://wiki.mozilla.org/Paladin/Misc/Android_Audio_Notes#12:00

 » automata joined the chat room. 12:20
 » cpearce joined the chat room. 12:34
 » automata left the chat room. 12:34
 » oneman left the chat room. 13:32
 » oneman joined the chat room. 13:49
 » sicking left the chat room. 15:08
 » sicking joined the chat room. 15:14
 » sicking left the chat room. 15:35
 » rillian_lime left the chat room. 15:35
 » rillian_lime joined the chat room. 15:35
 » sicking joined the chat room. 15:45
 » anant left the chat room. 16:49
 » sicking left the chat room. 16:55
jesup derf: I'll probably be on anyways, but is there anything you need from me at the codec meeting? 17:05
dmose jesup: is now a good time to chat about audio stuff? 17:11
jesup dmose: not horrible; the 3-year-old is watching videos on the tablet. Dangerous things, those tablets 17:13
dmose chuckles 17:14
dmose we're trying to figure out how soon to give ours access to such things 17:14
jesup Much easier for her to browse youtube. than a computer with a mouse. And with youtubve you have to watch 17:14
dmose true 17:14
jesup Good idea for a website: curated videos from youtube for kids 17:15
jesup the 11-month-old is "helping" unload the dishwasher 17:16
jesup Someone is making a ruggedized 7" (?) android tablet preloaded with stuff for kids - not out yet I think 17:16
jesup Flash on sesamestreet.org doesn't always work well with touch.... 17:17
jesup so, audio? 17:17
dmose good point on the sesamestreet 17:18
dmose on the upside, nice to see adobe acknowledging that flash is on its way out 17:18
dmose yeah, so you had mentioned last week after roc had filled me in on the current state of his audio work 17:19
jesup Yeah, in around 5 years, or more 17:19
dmose that you wanted to chat more 17:19
dmose so i was just responding to that 17:19
dmose i assumed/hoped you had some wisdom to impart :-) 17:20
jesup Right. I wanted to touch base with you on webrtc audio and how it might interact with some of the things you're doing 17:20
dmose ok 17:20
jesup We talked a little about it at AllHands 17:20
dmose yeah 17:20
dmose so, in general, i think the first person shooter use case is a pretty obvious one 17:20
jesup To lay the groundwork, what's the problem space you're dealing with/want to address? 17:21
jesup (I think I know, but better to make sure) 17:21
dmose primarily 3d gaming on the web 17:21
dmose https://wiki.mozilla.org/Paladin may be helpful 17:21
dmose and if you follow the (more…) link, that may be helpful too 17:22
jesup ok. (I'd looked a little before) 17:23
jesup So for audio, you have the typical in-game audio: background, music, and event-driven audio 17:24
dmose that sounds right to me 17:24
jesup And then possibly streaming or inter-player audio 17:24
dmose yeah 17:24
jesup I assume there's an assumption of multi-player, at least as an option 17:24
dmose eventually, yes 17:25
dmose that's not something we're thinking about much right now 17:25
dmose but it's clearly something the market wants for these games 17:25
jesup is the target "serious" heavy-duty 3d FPS type games, or casual-gamer (maybe 3d) games, or across the spectrum? 17:26
dmose long-term, across the spectrum 17:27
dmose but crawl, walk, run 17:27
dmose and we're also trying to care significantly about mobile 17:27
jesup gotcha. 17:27
jesup Ah. That will be.... fun 17:27
dmose so we're going to tend to pick our targets based on some combination of "what's most practical" * " most useful" 17:27
jesup Did we talk about the sound issues on Android? 17:27
dmose no we didn't 17:28
dmose i'd love to know more 17:28
jesup Well... it's bad. 1 sec 17:29
dmose chuckles 17:29
dmose good times! 17:29
 » lloyd|mbp is now known as lloyd. 17:29
 » lloyd left the chat room. 17:29
jesup http://mindtherobot.com/blog/555/android-audio-problems-hidden-limitations-and-opensl-es/ 17:31
jesup Google Android Audio Latency and you'll get a bunch of hits. 17:31
jesup Or look at Andoird bug 3434 17:31
dmose ah, that's super helpful 17:31
dmose t 17:31
dmose hanks 17:31
jesup Basically, latency sucks 17:31
jesup there are tricks to minimize it, but they don't always work, depending on many htings 17:32
kinetik i hope they've improved things in ICS. 17:32
dmose fun times 17:32
jesup http://www.kvraudio.com/forum/printview.php?t=331703&start=0 17:33
jesup "The only thing I recall is an article discussing ICS and the fact that there hadn't been any significant headway made. " 17:33
jesup  :-( 17:33
dmose that's unfortunate 17:33
jesup So, that's there. It's probably ok for games, thoiugh even there it requires work 17:36
jesup Music apps it sucks for 17:36
dmose good to know; i'm gonna go slap this on wiki page somewhere 17:36
jesup Interactive stuff like webrtc - not good, though we can probably work, with trouble 17:36
dmose that adds excitement 17:37
dmose presumably chrome will have the same issues though 17:37
dmose so perhaps that'll help motivate lower layer fixes 17:37
jesup There are tricks to improve it, especially if you can pre-load and prepare your sounds (games often do) 17:37
dmose yeah, we're going to need an asset-loading story 17:37
dmose since games actually real-time apps 17:37
jesup chrome may not, since it's the kernel drivers and flingers that are the issue 17:38
dmose can you unpack that a bit? 17:38
jesup chrome's kernel and androids are different; and I think the audio "flinger" layer is also. Some of the problems devolve from the interfaces for Java/Dalvik I understand. 17:39
jesup I'm no expert on this, but I've taken note iof it since my old company had been planning to build Android-based standalone videophones 17:39
dmose i don't mean chromeos 17:39
dmose i mean chrome-the-browser 17:39
dmose which is supposed to replace "browser" on android at some point 17:40
jesup Aha. Again, it's the kernel/flinger, not the browser that's the problem 17:40
kinetik (some stuff on android latency here: http://www.youtube.com/watch?v=gfiYUL2exT8#t=40m32s) 17:40
dmose right, but my point being that if google cares about webrtc in their browser on android 17:40
jesup Some of the hacks used by app developers for low-latency audio may be tough for a browser to do 17:40
dmose that may motivate them to prioritize fixing the kernel 17:40
dmose then again, maybe not :-) 17:41
jesup Yes. But they're in no rush for ebrtc on android last I knew 17:41
dmose kinetik: ah, thanks! 17:41
jesup kinetik: thanks! 17:41
jesup We spent a while discussing it at AllHands 17:41
dmose ah well 17:41
dmose too bad 17:41
jesup kinetik: any revelations since then? 17:41
jesup dmose: kinetik is another good person to talk to about this 17:42
jesup (audio in general, low-latency) 17:42
dmose good to know! i just started following the "areweplayingyet" tracking bug 17:42
kinetik jesup: not so far, i'm planning to start working on libcubeb backend for android shortly 17:43
kinetik and waiting for ICS to see what might've changed, latency wise 17:43
jesup As for Webrtc and gaming: one thing that may help you a bunch is the low-latency congestion control we're doing, at least for media streams. 17:43
jesup We're going to try to work it into the data streams too 17:43
dmose are data streams == PeerConnection? 17:44
rillian_lime they're a thing hanging off peerconnection objects 17:44
jesup Multiplayer can be enhanced with audio chat betwen players/teams/"break-rooms", etc 17:44
jesup yes 17:44
jesup direct p2p data channels (mostly, sometimes has to bounce off a relay) 17:45
dmose that'd be sweet 17:45
jesup We hope to support reliable streams and datagrams, unreliable datagrams, and possibly partly-reliable datagrams (limited retry) 17:45
dmose wow 17:45
dmose sounds like a lot of work 17:45
dmose but awesome! 17:45
jesup encrypted, congestion-controlled, with the app having input into who gets the bits 17:46
jesup (prioritized) 17:46
jesup We're leveraging SCTP -over-UDP (really over DTLS-over-UDP) 17:46
jesup (I assume, IETF consensus call on that tomorrow) 17:47
jesup My proposal 17:47
dmose break a leg :-) 17:47
jesup Most of that comes pretty much for free with SCTP 17:47
jesup The main things we're adding/improving is congestion cotnrol 17:47
dmose does the code we have already include SCTP? 17:48
jesup That's somewhat of my baby 17:48
dmose nice 17:48
jesup No, but we're working with the SCTP authors of the BSD implementation; they've split it out into user-space (actually updated the split) 17:48
dmose very cool! 17:48
dmose just waits for the day when firefox ships with a loadable kernel module 17:49
jesup Even ignoreing media, there's a lot you can leverage there 17:49
jesup b2g - inthe kernel 17:49
dmose fair point 17:49
jesup  :-) 17:49
jesup Webrtc and media - enables chat, group chat, locality-based chat with fading, messages, push-to-talk, etc 17:50
dmose yeah, there's a lot to like 17:50
dmose the interesting thing about gaming is that pretty much wants to inhale and use every possible capability of a given platform 17:51
dmose so we definitely have to pick our priorities :-) 17:51
jesup A useful demo/framework for some devs might be a 3-d meeting/chat/presentation with local audio between players "near" each other and a PA system. 17:51
jesup It's not actually a game, but shows how you can use the tech to make games, with owrking examples 17:52
dmose yeah, intriguing 17:52
jesup Maybe a "slideshow" as well, or enterpise bridge main display :-) 17:53
jesup and it removes the need to make the "game" portion, plus no one dings you on the game not being good as a game 17:53
dmose very intriguing 17:54
dmose does ircloggr have a way to show stuff in past->present order? 17:54
dmose i want to AIM some of the other paladin folks at the logs of this discussion 17:55
jesup I have to run in a sec, but another thing it will probably be able to do is to leave invites ("OFFERS") open with the server, and let it "fork" the OFFER to a play who has moved close enough so your browser and theirs can iopen direct audio channels 17:55
jesup It's logged - see title 17:55
jesup A bit painful, though 17:55
dmose yeah, i'm looking at those logs, and they're kinda hard to read because they're in backwards order 17:55
jesup If you're in chatzilla you can save the channel view 17:56
dmose yeah, i'll just do that 17:56
dmose that forking invites thing is a little brain-warping 17:56
jesup We wouldn't mind a good demo/example of what can be doen with it either 17:57
dmose duly noted 17:57
jesup Normally forking is when you call someone, and they are logged in via N different devices. The offer is forked to all of them 17:57
dmose ah, that's helpful context 17:57
jesup they all ring, and when one picks up the others are cancelled - but they don't have to be (think in a house wiht N phones) 17:58
jesup Or an offer to a meta-address for the workgroup, leading to a "mesh" conference (no central mixer). Trickier, but doable 17:58
dmose cool 17:59
jesup If they can remain open, the game server can leverage it. 17:59
jesup Gotta run 17:59
jesup hoep that helped 17:59
dmose ok, thanks for the brain dump 18:00
jesup ta 18:00
dmose that's super helpful 18:00
Retrieved from "https://wiki.mozilla.org/index.php?title=Paladin/Misc/Android_Audio_Notes&oldid=372429"