File:Brown bag - Panthea from Reboot.webm

TRANSCRIPT

PRESENTER: So welcome. Good morning, good afternoon, or evening, depending on where you are joining us. Thank you, Anna. So today, I'm really, really happy and pleased to be here with Panthea Lee. And Panthea is working with us in the movement strategy research. And she's going to tell us a lot more. Panthea is the principal of Reboot, which is a design research firm that has been partnering with us for now over two years, I think? Over a year. Wow.

PANTHEA LEE: We've done a lot of work in a year.

PRESENTER: Yes. We're really productive together. And I also want to mention that Zack is here with her. He's also principal at Reboot. And we hope to be learning a lot from them. And before Panthea starts talking, I just wanted to make sure that you all feel free to ask questions and interrupt and interact with us as you're hearing the many interesting, fascinating things that she's going to share with us today. OK?

So to start, tell us a bit more about Reboot.

PANTHEA LEE: So thank you for coming.

Hello? Great.

So Reboot is a design firm that works with mission-driven organizations to design policies and services and products that help them better meet the needs of their users and of their constituents. So that means we work with anyone from the UN to the city of New York to grassroots activist collectives.

And we were founded on the belief that people should have a greater say in the policies and the products that impact their lives. And we do that by bringing in a lot of the practices that the private sector uses to design things people actually want and need. And we try and translate that to the public sector, and to the social sector, where there's not always the same incentives or the same accountability mechanisms.

And so what that means in practice is we've done everything from helping the government of Libya post-revolution design the world's first mobile voter registration and elections management system to working with New York City on criminal justice reform. We've been working specifically on bail reform and helping immigrants better understand their rights. So, a range of things.

PRESENTER: Yes. Fascinating. Fascinating. And how the work with the Wikimedia Foundation, right? Like, how has Reboot been working with the Wikimedia Foundation?

PANTHEA LEE: Yeah. So, I think our work together started about just over a year ago. A lot of Reboot's work is global. And so I think it was at a point where the foundation was thinking about investing more deeply into understanding the really diverse communities that you guys serve all over the world. And so it really started with this New Readers project, where we were helping you all design and conduct design research in Nigeria and India to try and inform strategies from communications to product to partnerships, whatnot. And I know that's led to some really interesting things that Jack and Zack are doing around communications, around sort of offline support that Ann's leading, whatnot.

So that's where it started. And then we did some work around mapping your various audiences, developing an audience framework, and thinking about how to prioritize engagement and investment in research on different audiences, which led to some work around conducting design research with editors, actually. With editors? Great. And so we've been working with the editing team in mid-sized Wikis in South Korea, in the Czech Republic to understand the editor experience, and where all you are-- where you guys are gaining people and losing people, and learning from lateral communities, as well.

And then sort of coming full circle, we've been obviously doing work around movement strategy, Track D, now called New Voices, research in Brazil and Indonesia just to understand how are people getting information. What are the trends that the foundation and the movement need to be aware of looking out to the next 15 years. How do people get online. How do they come to seek and trust information online. What do they know about Wikipedia, and do they care. So some of those questions that we've been batting around to inform strategy process.

PRESENTER: And why design research? Like, how design research compares to a more analytics, data-driven approach?

PANTHEA LEE: You know, design research is at the heart of a lot of what Reboot does because we believe that fundamentally understanding human experiences, human behaviors, mental models is critical to designing things that people actually want and need and will use. And so what that means is, instead of just cold hard logic, looking at sort of market analytics, which are important, we also use empathy as a primary processing tool. And it's been really cool to have foundation staff with us on all of these research projects to understand, OK, once we talk to all these different users and potential users of Wikipedia, can we walk in their shoes. Can we understand what they care about, how they feel about things, how they get online, what the challenges that they have are. Because if we can understand that and put ourselves in their shoes, that will make us better strategists. That will make us better designers.

And I think it's actually-- we do a lot of generative design research, which means we don't necessarily know what the solution or answer will be at the very start of the process, compared to some of the more I think evaluative work that the foundation currently does, which is also very important. I know the product teams have been doing a lot around sort of testing different sort of products and approaches and iterating upon those. But I think some of the more generative work here has been really valuable for this sort of strategy process where we want to think bigger picture.

PRESENTER: And you were thinking-- we were just talking about the movement strategy, and how ambitious this big process is. And I think that everyone is really interested and excited to hear what are the findings so far from Brazil and Indonesia.

PANTHEA LEE: We have a 25-page memo that you're all welcome to read. You know, I think we have a lot of findings that we're still processing and want to work with you all and the movement strategy team over the coming few weeks. But I think one of the biggest things that is coming out from the research is that I think, looking forward, Wikipedia and the Wikimedia movement is going to have to think about how Wikipedia is not just a destination for knowledge and for information, but how it can become a source of knowledge and a source of information in all the different and diverse ways that people learn.

And what I mean by that is, you know, the internet was a very different place when Wikipedia first started. What we're seeing now and what we're finding through the research is people are learning and getting information in all these really diverse and fascinating ways, from people learning to cook and groom their eyebrows on YouTube to all the different homework help communities, the Brainly.coms, the other things. You know, and young people especially are just finding really creative and new ways to learn, to share information, to get information. And they're not really going to websites anymore.

And so, you know, if we play that out for Wikipedia, I think it's interesting to think about how the site and how the platform and all the knowledge that you guys have, how to make it more modular, how to make it more portable, to be able to take the resource that you all have built and feed it into all the diverse and different channels in ways that people are learning.

And I think that's going to be something quite interesting because, you know, the first 15 years was, I think, for you all really about building this incredible resource, and really thinking about how this production model works. The next 15 years might be really thinking about, OK, so then once we have this information, how do we actually get it out to people. What is the distribution model. And I think that's going to be really interesting to look at how Wikimedia innovates next.

PRESENTER: And some of-- and what would be some of the key insights when you're thinking about, like, Brazil and Indonesia specifically? Are there things that stood out for you?

PANTHEA LEE: Something that was really fascinating for us was looking at the rise of messaging apps, and just how prevalent and popular they are now. I think WhatsApp is installed in something like 2/3 of smartphones in Indonesia. And it grew 300% between 2015 and 2016. And there's a lot of factors driving that, one of them being that a lot of telcos, mobile network operators, are offering these apps for free. They're zero rating them, or they're including them in sort of data bundles and packages and whatnot.

And so these are really-- you know, whether it's social networking apps, messaging apps, they're really becoming people's onramp to the internet. And for some people, they are the only ways that they are getting online. And they're not even thinking about using WhatsApp as being online, or using the internet. And where cost is a factor or where it's just, like, fun to chat with and share links with your friends, what does that mean for Wikipedia? You know, how--

And I think what's interesting, as well, is people are also using these messaging apps now not just to chat and to share links. They're also forming these what we were calling sort of hyper-targeted social networks. Like, people don't want to use Facebook anymore. They're saying, gosh, Facebook is for old people. You know? We are-- I mean, I was like, OK, I use Facebook.

But so Facebook is for old people, what we want to do now is we want to form our networks for, when we go to an event, afterwards we have our community that we form a WhatsApp group around that. I have a study group for every single one of my classes at university. I want to form these hyper-targeted social networks, and then-- and my free messaging app that telco is supporting is enabling me to do that.

So what does that mean for Wikipedia? How do you guys appear and show up and help people in these study groups, and appear organically in context? Is that in-app previews of Wikipedia content? Is it some other ways to sort of help people share information where and how they're doing so? And I think that was really fascinating to us.

And then I know you guys know this, but there's obviously a lot of brand confusion that we surfaced. And I know it's been talked about, and I know the comms team is working on this. But Wikipedia has great brand recognition, but a lot of brand confusion. And so what that means is people think of you all as, you know, this is a technology giant. We know the name, but we'll compare it to a Google or a Facebook. That's kind of crappy for you, because you get judged then to be a pretty poor search engine or a really confusing social network.

And so, you know, how do we help people really understand what Wikipedia is, how it works, and why they should care. And those are some of the things that are coming out. I'm happy to talk about any of the other findings that surfaced.

PRESENTER: Yeah. And I'm also interested, like, in talking about the youth, right? Like, you mentioned that, and what are some of the key interesting things that you're seeing particularly with that audience, and if there are things there that we should be particularly paying attention to as we think about our next 15 years as a movement.

PANTHEA LEE: That's a big question. What are the kids doing? So I think one thing that is really interesting about the youth that we were looking at is trust in content-- whether or not they trust content doesn't really matter. Trust doesn't equal usage. Trust doesn't equal utility. You're seeing a generation that is highly skeptical of media, of information sources, and particularly of online content. And there's a lot of factors that give rise to that.

In Brazil and Indonesia specifically, you have a long history of government control of the media. You have highly concentrated media ownership in both markets. You have the proliferation of, you know, fake news, and people now use sort of fake news as a term to describe seemingly everything. And you have business models that incentivize clickbaits and sensational content. So there's lots of reasons that we can talk about.

But ultimately, what we're seeing is young people don't trust the content that they have, or they know content is biased, but they will use it anyway. They will then take that content and discuss it with their friends to try and triangulate between lots of different sources to say, OK, you know, should I use this, for what purposes, I know it's biased. And I think that's quite interesting, because Wikipedia and Wikimedia spend a lot of time thinking about trust and accuracy. And that's great, but also how do we think about relevance and utility, and what's actually going to get people to use this content.

And you know, one of the things that we're seeing, too, around trust is that the indicators of trust are changing. You know, young people are not trusting institutions to give them credible, verifiable content. They're trusting each other. And so they are looking for indicators such as number of followers, number of likes on articles, other sort of social, more individual indicators-- you know, the reputation of a content curator-- to help them determine what content they should trust and use.

And I think there's really interesting implications there, then, for Wikipedia, because your process and your content is driven by individuals. And so is there a way of showcasing that and surfacing that to help people understand how the sausage is made, because they want to know that to then understand whether or not they should invest in and use a piece of content.

Yeah, and then maybe the final point on young people is that, you know, I think we hear visual, we hear real-time, we hear social, and we know all that. But those aren't just buzzwords. I mean, kids, young people are getting their news on Instagram. You know, instead of going to the website of a newspaper, they are following the Instagram account of a newspaper, because we're sitting there and they're scrolling through like this is the exact amount of content that I want on a significant news story. Less than 100 words. That's it. You know, big photo, great. Scroll through. They're getting breaking news by following trending topics on Twitter. They're then taking that to WhatsApp to discuss with their friends.

So what does that mean for Wikipedia? You know, do we need to think about using-- allowing video as references, for example. Do we think about, I don't know, push alerts around articles that are getting sort of rapid distributed and concentrated edits so people-- you know, so Wikipedia can be seen as relevant and real-time in the way that they want their news and information. I think those are interesting questions to explore and wrestle with.

PRESENTER: Yeah. And I'm wondering here, now that you just mentioned some of the possible opportunities and things that we could be exploring and considering, and how we take all that, right. We touched base on-- you touched base on some of the bigger key findings how youth is relating to content information, how they're accessing it. So making sense of that and connecting that with the five thematic directions of the strategy, like, how are you seeing those? How are you seeing what you learn and saw from Indonesia and Brazil relating to the teams?

PANTHEA LEE: So, we were really excited to see the five movement strategy themes. They are ambitious. They are visionary. They are comprehensive. And we were then also thinking about how to map our findings against those. And I think what's interesting is we try to separate out between objectives and strategies and then tactics, because I think the five themes are slightly different in that way. So I'm going to get the letters confused.

PRESENTER: I have them.

PANTHEA LEE: OK, you have them. But you know, the ones around being a respected and relevant source of knowledge, that is an objective. And we can do that through-- by advancing with technology. And we can do that by engaging the knowledge ecosystem. Those are tactics. You know, becoming a truly movement, that's an objective. And you know, again, we can do that sort of through technology, through engaging diverse partners.

And so as-- perfect. And so as we were mapping some of the opportunities that we saw-- and I know we have a workshop with some of the Movement Strategy folks after this-- we tried to essentially sort of map our opportunities into a matrix of sorts-- I know folks can't see this-- to understand how we use technology to meet each of these objectives. Obviously, the work that we were doing was really focused on readers, on communities, on audiences. But we know that, you know, just strong, healthy communities will be critical and foundational to driving all of this forward. But you know, our work under this wasn't really focused on Track A.

And I think what's really then interesting about this is this notion of being more modular, more portable, you know, engaging with diverse partners to push content out, I think that's really going to be at the intersection of all of these themes. You know, how do you work with educational institutions to think about using Wikipedia content in after-school programs that extend learning outside of the classroom. How do you all work with nonprofits that are investing in skills training for unemployed youth, to take Wikipedia content, mix and match it to develop a curricula.

I think those will require both a mix of technology and partnerships. And I think, you know, those will effectively help Wikipedia be relevant and global and sustainable into the future. So, yes.

PRESENTER: Thank you. Thank you for showing and talking about this map, and how you are integrating that based on the data. And I think another thing that I would love to hear from you is that how do you think we are serving the emerging markets and emerging communities? Are we-- like, what are the things that we could be doing differently there? And what are the ways of actually identifying their needs and then serving those needs?

PANTHEA LEE: So I think, in terms of emerging markets-- you know, I think one thing that I know we've all been talking about is emerging markets, global [INAUDIBLE], it's very broad term. And you all have been investing in research in a lot of specific markets. So, you know, I think one key step moving forward is to try and disentangle some of this, and to break these markets apart.

So if you take a look at Indonesia and Nigeria, for example, where you guys have invested in primary research, these are quite different markets. Let's take a look at mobile penetration and cost of data, for example. Nigeria may be representative of markets in sub-Saharan Africa. You know, data is relatively expensive, and so people ration it. It's kind of a scarce resource. And so the strategies that you guys develop there will be applicable to certain regions.

But Indonesia, where cost of data is dropping quite rapidly, even though there's still sort of barriers to access, that might be a much more sort of illustrative view of where different markets are going in terms of internet usage, information behaviors, whatnot. And so I think first step might be to sort of disentangle emerging markets, and think about what each market can tell you about larger patterns in which regions.

And then I think, from there, we see that Wikipedia has-- we talked about sort of brand recognition, whatnot. But I think one group, or one set of users that the foundation I know-- and the movement its thinking about whether and how we can better serve them is what we might term sort of more marginalized communities, lower income users, whatnot. I know that's been a topic of conversation here.

And I think that is an area where there is still a significant gap, through no fault of the movement. You know, these are populations that, for lots of reasons-- you know, levels of education, income, whatnot-- simply are not getting online, or are not able to get to and use the information that Wikipedia provides once they are. And so I think an interesting question in these markets and with these users in particular is what-- does the movement want to serve them, and, if you do, who are the partners that you might need to engage to do that.

So there are nonprofits, there are government programs, there are other folks that sort of specialize in serving communities and populations such as these that very much need the content that Wikipedia has, that could really learn from and draw on the energy and the passion of the community that you all have. These folks-- you know, we work with tons of nonprofits [INAUDIBLE] we want to develop educational content in x or y way. We have the-- we know how to reach low-income populations, but we don't necessarily know how to package and put together things for them or to have the resources to invest in developing this content. So I think that's an area that the movement might continue to explore, because we do see that as a gap right now.

PRESENTER: So I think we already-- I don't know, I don't have my phone on me, so I don't know how we're doing on time. 10:30? How are you folks feeling in terms of asking questions and jumping in? Because I think I have more questions, and I can keep asking them, but I just want to do a check and open a little bit more of the floor for us to have more of a conversation with Panthea and with the group.

So, how are we feeling? Can-- like, do we have questions, do we have comments? Yes, we have Sati there. Brendan? Go over there. Oh, there's no mic there. OK.

AUDIENCE: Hello. Hello? OK. So, I have many questions, too, but I'm going to ask one that's probably not fully filled out. And I guess my question is a lot of what you've said have been talking about how-- kind of a comparison, right? Like, we sitting here today in San Francisco understand a world a certain way. We see Wikipedia in a certain way. We do research, and we ask people to understand kind of Wikipedia in their own context, right? And then there is this feeling I guess I get where we say, well, they have barriers to, let's say, accessing the world in the way we do.

And I guess I'm asking, that's very-- that focuses a lot on deficiencies, right? Like barriers, like things that they don't have, or ways in which they do different things. And I guess I'm asking, in what ways should we actually be learning from them? Should we actually be saying maybe the world is trending in a way where we're actually behind and they're progressing, and instead of trying to match them to us, maybe we should be matching us to them. Does that make sense?

PANTHEA LEE: Yeah, no, absolutely. I think that-- I don't necessarily mean to say, you know, their people have deficiencies. I think it's more there are barriers. So Wikipedia works a certain way right now that then sort of erects barriers to people being able to access and use you all. And so, you know, what are ways that then the Movement Foundation might think of addressing those barriers. I do think that there are things that the movement can definitely learn from the creativity of-- and it's hard to talk about these populations, because we're talking about quite a diverse set of users and markets.

But I think one of the things that has been really interesting is the rise of these social networks. And people that have cost barriers to be able to use the internet are getting around them in quite creative ways. You know, when telcos are offering a file transfer for free on these, all of a sudden you see a rush-- you know, everyone is now, instead of using email, just using, like, WhatsApp to basically send everything around. And so that's actually quite interesting. Instead of using other social networks, they are creating their own. And I think that's a really interesting way for people-- for you all to explore.

I think also there are really interesting-- you know, we work a lot with nonprofits and media organizations and whatnot that are leveraging traditional media in quite interesting ways, as well, thinking about how do we use television and radio in creative ways to deliver educational content. And I think that could be an area for you all to explore, as well.

AUDIENCE: So then I guess my follow-up to that is then you've kind of laid out, it feels like, two models, one where, right now, we're kind of this self-encased thing, right? A website, like a Facebook, like an ecosystem, in a sense. And what I'm hearing from you is that we actually need to be more ubiquitous, right? That we need to be embedded in some way, we need to show up in ways that might not even be branded in a way that's recognizable. But if our true mission is about knowledge-- let's say a part of it is about knowledge dissemination, then that knowledge, it's really important for that to show up in the ways that people are accessing it.

And so how, I guess, in that vein-- and that's a lot about readers-- how do you think about content, let's say, curation, creation? Kind of the upstream pieces to dissemination. How does that end of the flow match into kind of this ubiquity that you're kind of describing at the other end of the flow?

PANTHEA LEE: So I think that-- yeah, I think you're absolutely right. Instead of becoming like a product, or instead of thinking of ourselves as a product and knowledge here, how do we actually be, like, an engine that pushes information out in ways to enable ubiquity, to be able to push it out to other content creators or folks that are providing educational resources, whatnot. And I think that the role of curators here is really interesting and important.

We see a lot of young people following vloggers and bloggers and, you know, that have built up trust in certain ways that may not see Wikipedia content as credible or easy to use or whatnot. And so does the role of the community, for example, change here? Do the role of the affiliates change? You know, right now I think the community often-- in the way that I understand it, a lot of it is dominated by editors that are contributing content and doing a lot of different functions.

But you know, in the future, is it about developing topic guides to help nonprofits that are thinking about digital literacy. OK, so these are the resources that we have, these are-- you know, through APIs, through more modular content, whatnot. This is how you might be able to mix and match it to design your own curricula in x or y way. Is it about thinking about community members as liaisons between other content creators, whether they're more individuals-- you know, alternative history vloggers on YouTube, to nonprofits focusing on educating people about women's rights. I think those are different ways that you all might be able to think about it. But I think this ubiquity point is a really interesting one, and definitely worth considering. Please.

AUDIENCE: It's a really great question where you ended. And I think one other piece of it that's somewhat simple to add on to what Panthea said is the more ubiquitous you are and the more people you're reaching, the more opportunities you have to build somebody up the ladder of engagements where they might become a contributor and actually be an editor. So there is a sort of feedback loop that comes from reaching more people in terms of potentially increasing the pool of contributors.

PRESENTER: Oh, yeah. So let's maybe go over them.

AUDIENCE: I think that you should stack the first question.

PRESENTER: Can you--

AUDIENCE: Yeah. Let's see. Let's get some people. All right, so we have a set of questions from blue jeans here. Jamo, you're first on the list. And then we can go to Edward and Amir.

AUDIENCE: Hello. How's my audio?

AUDIENCE: You sound great.

AUDIENCE: Aw, thank you. So, thank you very much for speaking with us. I had a question about methods because I'm a design researcher, and so I like methods. My question is around how Reboot gets to the point of providing the high-priority findings and recommendations. So I believe that a lot of people have an idea of how design research works in terms of, you know, you talk to people, you observe what they do, you go to where they live, you find out kind of how they interact with, in our case, you know, Wikipedia or information technologies or technology in general, how it fits into their lives. And then you take lots of notes.

But I think that it's really interesting and kind of less understood how you get from, OK, we learned all this stuff about people and about what they do and what they believe and what motivates them, how do we get to the point where we are synthesizing that and prioritizing what are the most important findings? What are the big takeaways for this audience? You know, for our clients. Wikimedia, in this case. I think we did a really great job of that in the New Readers project, but I'd love to hear a little more kind of about how that process works within your organization.

PANTHEA LEE: I love nerding out on methods. And so thank you for the question. You know, so I think for us-- I'll try not to get lost in details, but I think, you know, where we started with you all is thinking about this research framework, and what were the questions that we wanted to ask. And usually, where we start here is one principle we have in mind is always don't necessarily ask about the thing that you're most interested in, which means we did not start by asking people about Wikipedia. We started by asking people about just information systems, like what information they need, what they care about, how they get it. And we go from sort of the broadest to how they get information to then sort of narrow it how do they get online to then how do they come to Wikipedia, how do they use Wikipedia. So we go from sort of broadest to most narrow.

Happy to talk about the research framework. I know that's sort of been shared with different team members. And then as part of the field research, we bring a team of local researchers from the city, the community, wherever that we're working, and then paired with a Wikimedia team, as well, to make sure that we both have the institutional sort of context and knowledge, and then also the sort of local and cultural translation, as well.

For us, it's really important to make sure we spend a lot of time with you all to understand your strategies, your work processes, whatnot, because I think sometimes user-centered design is misinterpreted as, you know, just let's focus on the end user, whereas in fact, you know, we actually need to do a lot of work to understand the organizations and the institutions that serve them to be able to design strategies that are actually feasible and implementable, rather than a shiny blue sky deck that you can't do anything with.

And then I think from there, in terms of the actual research, we do a lot of sort of semi-structured ethnographic interviews. We do user observation. We do tech demos, whatnot. I'm happy to get into any of those. But then we end up doing nightly synthesis sessions with the entire team, which means we are making sense of the data collected every single day. What are the patterns, what are the connections, and where should we go deeper the next day, because this type of research, it's really applied. And so we're not asking the same set of questions every single day. If we're asking the same questions we did on day 10 as we are on day one, we would have failed. You know, as we're thinking about opportunities, we're trying to figure out how to hone in deeper on those so that we can start sort of testing some of your hypotheses. And so it's quite of active and iterative research.

And then I would say, in terms of how we got to some of these higher priority findings-- and then I'll stop because I might be losing people-- is we tried to map all the users that we spoke to in each context against a matrix along sort of digital competence and literacy, and then also the type of use cases that they are using the internet for. From there, we then segmented into a couple of subsets of users. And so you all have user personas now for these sort of different archetypes. And then from there, we mapped the user journey of each of these types of users to help us understand, in their information journey to try and get the information that they want and need to do x or y, depending on the type of user, what are the biggest barriers that they face and what are the most valuable information sources for them.

For the most valuable information sources, we then think about what can we learn from these lateral examples, and then for the biggest barriers we then try and sort of aggregate them to understand, OK, what are the most prominent and significant barriers that are preventing more users from being able to take advantage of Wikipedia. And then that's how we end up rank ordering the opportunities that then we put forward to you guys.

That was pretty fast. I'm not sure if that was useful.

AUDIENCE: Excellent answer. Thanks. Nice work. OK, so we had I think Edward next, is that right? Edward, would you like to take over?

EDWARD: Sure. My apologies, I'm in a coffee shop right now. So can you all hear me?

AUDIENCE: We can hear you.

EDWARD: Thank you. Yeah, so thank you so much for talking with us. Yeah, this is super interesting work, and I've been following the audience's work for a while. And I've actually been trying to use it in my own work. So I do a lot of surveys at the foundation and with communities, so I'm also interested in the methods a little bit. So I am actually curious to hear generally what have been some of the limitations or challenges that you're finding in conducting your research, especially what you said at the beginning, which is around how you try to bring voices into the design process. So what have been some challenges around that? And perhaps if there's been anything in our context that has been challenging for you.

PANTHEA LEE: Time? Challenges, I think that one of the biggest challenges for us has been thinking about the breadth of opportunities there, and then how to organize them, which is why sort of-- going to sort of Jonathan's question previously, for us having a really structured process, to narrow down, to rank order to process the opportunities and findings was really important.

But I think that has been a challenge, but I think what it's been helped by is the fact that it's been really refreshing and really exciting to work with such a cross-functional team. We've been working with Global Reach and partnerships and community and community engagement and comms and product and whatnot. And so you all having the conversations to then give us sort of a more targeted brief, that's been really useful.

I think that we've-- I think there is more that we could do to think about how to use surveys and other sort of quantitative methods in conjunction with the more qualitative methods that we use. Typically, when we do this, we-- design research is really good at going deep and understanding people's behaviors, attitudes, whatnot. And I know we've-- and we've been working with Dan and taking a look at some of the trends and findings from the mobile surveys, thinking about how that informs our research. I think we could do more perhaps to think about, OK, so now we've gone really deep, how do we go broad again to test out the representativeness of some of our findings here against a wider audience.

And then I know that I think one area that has also been somewhat challenging is thinking about-- is the time factor. You know, we've been doing this sort of quite rapidly. And we've had basically time for about two-week sprints in each of these countries. And so-- which has been about sort of 70 respondents per country. And so we haven't really had time to be able to prototype any sort of solutions or strategies, which sometimes, given a longer sprint, we are able to do to test out and come back with user feedback on specific strategies or product designs that you might want to pursue.

PRESENTER: On that note of time, I think there was a question also there about the user personas not being posted. They're not posted yet, but they will be soon. So there's a big DAC coming with all the personas from Indonesia and Brazil, and more information. So yeah, stay tuned for that. And then I believe that the next question is from Amir. Is that right?

AUDIENCE: Hello. Can anybody hear me?

PRESENTER: Yes, we can hear you.

AUDIENCE: Oh. You might also hear my son in the back. OK. So, quick question. It was all really, really interesting. Thank you so much for coming and talking about this. I appreciate this a lot. The global perspective is really important for us. My question is about something that you mentioned in the beginning of your talk. You said that the messenger networks like WhatsApp are really important today. How do you think Wikipedia could get there? Because currently, we are very much a web organization, and people access us through browsers. And a few people access us through the Wikipedia app, but it's almost the same as reading it in the browser. What could we do with these networks, given that they are so important and popular, possibly even more popular than Facebook by now? What could we do there?

PANTHEA LEE: That's a good question. So I think that first of all it's just I think really understanding what people are using these networks for. And so, you know, it wasn't really a focus of our research, but it sort of emerged quite quickly as a key trend that we are seeing. So, you know, I think I mentioned, you know, whether it was study groups, I think that is actually-- that could be a big opportunity for Wikipedia in terms of people are talking about homework assignments, they're debating topics learned in class and whatnot sort of on these networks. And is there a way to bring in Wikipedia to be sort of like a context provider when people are having these debates and conversations.

I'm not sure of the specific sort of product strategy that would mean, but basically how do you be sort of organic and in context. I think there's been some experiments with WhatsApp chat bots? I'm not sure. I think I heard something about this. But those could be ways to help people, you know, again, sort of get information from Wikipedia that is, in many markets, you know, low-cost and where people are.

I think that there's also-- in many of these markets, we found that the ways that-- so this now maybe ties a little bit into the editor's work that we've been doing, but in some of these markets, there are not great representations of, like, essentially sort of people's, like, culture in context on the internet. I think Wikipedia's current model of, you know, determining sort of what is good content that can be included on the site presents some barriers to cultures that may be sort of more oral or may not have the resources and the references, rather, to be, like, you know, a verified and trustworthy sort of Wikipedia article.

Are there ways to have different versions of Wikipedia, perhaps, that allow different types of contribution that you might be sourcing via these chat channels that then sort of gets put into larger processes of verification and whatnot. Do you guys think about it as a way to sort of get content, or just source material that then might be put through more rigorous processes. I think those are also things that you could explore. But yeah, those are some ideas.

PRESENTER: Amir, does that answer your question? Do you have follow-ups or comments?

AUDIENCE: Yeah, it answers the question. Just a tiny follow-up. You only mentioned WhatsApp, but I know that there are several other platforms around the world, like Viber or WeChat or Telegram. Are there any others that you suggest looking closely at? Which are the important ones around the world, or maybe in particular regions?

PANTHEA LEE: I know some of this is in our findings memo. I do know, in both markets, WhatsApp was dominant. I know Line and Telegram are-- and people use them for various purposes. You know, we've heard WhatsApp we might use for more sort of professional and schools reasons, but we might use Line because we like their emojis better. And so we didn't really do a deep dive into comparing the messaging apps, but there is good market research on some of this.

A note of caution on the market research is oftentimes, they really sort of just give you penetration in terms of, like, downloads, whatnot. And that may not be the same thing as usage. And so I think sort of probing into why people are using specific messaging platforms, for what and how could be something that you guys might want to think about further. Yeah.

PRESENTER: All right. We have a question there. Thank you, Amir. And we have Rosie there with a question. Do we have the mic in the back?

AUDIENCE: Thank you. This has been pretty enlightening for me. I have a question about who's being left behind. So for the first 15 years, there's been a lot said, a lot written about the teenagers and young men in their 20s are the ones who kind of dug in deep and were the ones that spent the most time on the Wikipedia that we've known of the last 15 years or so, and that there were definitely big segments of people who were kind of left out. A very small percentage of women versus men, and so on.

And so I'm wondering, does your research touch on that? Do you have a feeling for-- and you've spoken to groups of people in Indonesia and Brazil, but can you sense who has been left out of the conversation in Brazil, in Indonesia? And is there a way that, in the next 15 years, we're not going to replicate this feeling of somebody being left out? Will we be able to be more inclusive based on the things you're kind of learning so that we kind of learn from what we've experienced in these first 15 years?

PANTHEA LEE: That's a really interesting question. So I think, on just your earlier point around it being mostly young men really digging into this, I think that our work on editors, which has been in South Korea and Czech Republic, that somewhat confirms what you're saying here. And we were sort of probing into where you lose people in terms of contributors. What is the editor experience, and where do people drop off.

And I think we see that there are elements of policies, norms, cultural norms and practices that, you know, pose barriers to, I think-- you know, we saw sometimes, you know, women feeling excluded, or just it was a difficult-- I think we heard people say, you know, Wikipedia is almost more hassle than it's worth, getting into the tussles to be able to contribute. And so I think we saw that-- I'm not sure-- and I think now people are then finding and founding other-- you know, we saw FemiWiki in South Korea, this new resource that had been developed by contributors to Wikipedia that just didn't want to get into the fight, as they saw it.

We're seeing sort of older people, retirees that we spoke with that chose instead to contribute to other platforms. We were analyzing how Coursera, the sort of MOOC platform, how they recruit and, you know, onboard and then continue encouraging contributors to their platforms. And they've been successful we've seen with retirees that actually have a lot to give. And so those are some of the things that we're finding through the editors research.

In terms of movement strategy, I think we got a sort of big push from Adele and her team really to look at who's been left behind, and to really talk with populations that are not currently online, lower income populations, to test out some of our hypotheses that, you know, Wikipedia may not reaching them and may not be able to on its current trajectory. And I think that's really where some of the ideas around partnering with nonprofits or other social organizations that really serve these populations and who's, like, you know, core value is doing outreach to them, and then thinking about how you guys bring your unique model to then be the engine of their work in terms of contributing content or, you know, community members, helping them navigate your content, I think that's where some of those ideas came from, out of the push to speak with those types of populations.

PRESENTER: Absolutely. Yeah, and I think for us, when we were thinking about this and, like, really taking advantage of design research and really being able to hear from the users that we were trying to serve, we had no limits, right? It was really like let's go there and really understand. And I think we were trying to probe and go further into who we are leaving behind now. So those are really, like, a lot of people in this countries, right? And they are, like, women, not nonwhite populations, right. Like, so racially and ethnicity diverse populations. Age diversity. Like, all the things that we have-- that we feel that we should be paying more attention to.

And those are part of the new voices that we feel that have not been represented or included in our conversations, right. And now, in this process, we want them to be sitting at the table and being part of that. And I think the work that we did in Brazil and Indonesia, but also with all the other work that we have been doing in all these countries, were really our go to that. Let's really open the floor, and let's really elevate those voices and make sure that we're hearing from them.

And we are now at cycle three, right, in the movement strategy process. And I think that is how this information is coming right to the communities. And we're sharing all that. And we are connecting, OK? So where we want to be in 15 years, and can we work and have those new voices as part of this movement, right? Can we build something that it's not-- that is with them.

And I think we are-- like, I'm excited for this phase, because I think we have a lot to exchange, a lot to learn from each other, right. Like Sati was talking about, how we learn from them. And I think there's so many things that we have to learn. And they are out there, right? Like those findings, the insights, and really interesting creative ideas are out there. But we need to open up a space for the new voices to be in the dialogue. Yeah.

One more question. And who is it? Jessica. Jessica, are you there?

JESSICA: Yeah. Can you hear me?

PRESENTER: Yes, we can hear you.

JESSICA: Perfect. Thank you. Thank you for this great talk, to begin with. It was really, really, really interesting. Just a quick question. And I know we don't have a lot of time, but you mentioned something that was really interesting, I thought, around brand awareness and brand confusion, that a lot of-- about-- lot of people know about Wikipedia, but they might not have a good understanding of who we are and how we work. And I was just interested in getting some high-level thoughts and suggestions from you of, you know, how that can be addressed. What can we do better as a movement and a foundation side in order to educate people about who we are and how we work and why they should care, as you put it very clearly?

PANTHEA LEE: So, I think that we're seeing a lot of interest from different users around sort of understanding the process behind, like, how the sausage gets made. And we heard people say, you know, I know Wikipedia, but they seem to be-- they're not very transparent. We don't know where they are. We don't know how they work. But you know, Google seems much more transparent, because Google posts videos about, you know, how they work. And we, like, see their offices, and their offices are colorful. And we're like, really? That's so fascinating.

And so, you know, and we saw people-- we had users that were on the Wikipedia Instagram. We were watching them sort of scroll through, and they go, I don't get it. Like, what's the logic, basically, behind all of this content? You know, because people are-- you guys cover everything. And so people are following Instagram accounts that basically relate to a specific theme, you know, like cute cats, beautiful sunsets, you know, space, whatever. And so, you know, they're trying to understand also, like, what is Wikipedia, because I can't make sense of all of this content.

And so, you know, I think there are things that you all might be able to do to expose the process. Who are the people behind Wikipedia? People want to know. People want to know how Wikipedia is made, who are the people behind them, because they're trusting-- like, we're seeing trust shift from institutions to individuals. And so you guys are a movement of individuals. Let's show that, because-- and there are things that you can do, whether in terms of communications campaigns, but then also I think on articles and on the platform to be able to show that process to help people understand how it gets made.

And then I think around the actual sort of product, is it around just aggregating, you know, just knowledge, you know, one platform, and developing subchannels, topic guides, whatever it is, thinking about how you communicate, whether it's subchannels around your Instagram accounts or is there other things to help people understand Wikipedia is all of these things, but I don't have to just engage with Wikipedia as, like, one singular platform. You know, I can find whatever it is that Wikipedia offers that is interesting to me and sort of ignore the other stuff. And I think that could be another interesting area to explore.

But I know the comms team has been doing different sort of communications campaign in local languages and through other sort of locally relevant distribution channels. And so they might be better suited to speak to that.

PRESENTER: So I think that was our last question. Thank you, Jessica. Thank you, Panthea. This was really good. We're really happy to have you here. And yeah, thank you for all attending, as well. And that's it for this morning. Thank you.

PANTHEA LEE: Thanks for having me. This has been really fun work.