Over-the-counter data

Diagram of over-the-counter data components
Diagram of OTCD components

Over-the-counter data (OTCD) is a design approach used in data systems, particularly educational technology data systems, in order to increase the accuracy of users' data analyses by better reporting data.[1] The approach involves adhering to standards that are organized by five components: Label, Supplemental Documentation, Help System, Package/Display, and Content.[2]

OTCD was inspired by the varied ways over-the-counter medication supports those using its contents.[3] Just as it would be negligent for over-the-counter medication to contain no labeling, documentation, or other supports helping people to use its contents safely, it is deemed negligent for data systems to display data for educators without providing them with the necessary supports to best ensure it is used correctly when educators use the data to treat students’ needs.

Background

Inspired by the varied ways over-the-counter medication supports those using its contents, OTCD was created in 2010 and applied to the improvement of education data systems.[4] Consider the way in which the Food and Drug Administration (FDA) requires over-the-counter medication to be accompanied by textual guidance proven to improve its use, deeming it negligent to do otherwise.[5] With such guidance, patients may take over-the-counter medication with the goal of improving wellbeing while a doctor is not present to explain how to use the medication. No or poor medication labels have resulted in many errors and tragedy, as people are left with no way to know how to use the contents wisely.[6]

Labeling conventions can translate to improved understanding on non-medication products, as well.[7][8] Thus, in the way over-the-counter medicine’s proper use is communicated with a thorough label and added documentation, a data system used to analyze student performance can include components to help users better comprehend the data it contains.[9] Using an OTCD approach (i.e., following OTCD Standards) when communicating data involves following research-based recommendations likely to improve educators’ understanding, analysis, and use of the data being displayed.[10]

OTCD Component Appearance in Over-the-Counter Medication Appearance in Data Systems & Their Reports
Label The container label provides the name and info to questions like, "How many should I take?" and "What are the possible side effects?", etc. The report has a clear and concise title, and included in the footer or side are annotations that provide info most relevant and important to the report.
Supplemental Documentation Not all the info a user needs to know can fit on the label, so a folded-up piece of paper is enclosed within the package to offer further explanation. Similarly, explanatory info can accompany each report via links to a reference sheet and reference guide specific to each report.
Help System Users want an online help system to explore and discuss specific questions (50 million people use WebMD every year[11]). An online help system can offer comprehensive lessons on using the system and on data analysis (specific to the data).
Package/Display How the product is displayed and packaged helps communicate by clearly identifying the most important info, such as purpose and use. How data is organized and displayed, such as layout that encourages correct analyses for each particular report, helps to avoid confusion.
Content The ingredients of the product are vital; they have to be effective, user-appropriate, and not expired. The contents of each report and the report suite as a whole are effective, audience-appropriate and not expired.

Nonetheless, labeling and tools within data systems to assist analyses are uncommon, even though most educators analyze data alone.[12] Essentially, data systems and reports do not commonly present data in an “over-the-counter” format for educators, whose primary option for using data to treat students is thus compared to ingesting medicine from an unmarked or marginally marked container. Just as it would be negligent for over-the-counter medicine to contain no labeling, documentation, or other supports helping people to use its contents safely, it is negligent for data systems and reports to display data for educators without providing necessary supports to best ensure the data is used appropriately and thus has a desirable impact on students.[13]

The recommendations summarized by OTCD Standards (below) are based on research in education and edtech, as well as research in a variety of other fields (e.g., behavioral economics, design, business analytics, technology, and more). An OTCD approach is not meant to replace educators’ professional development or other interventions that improve data use, but it is an added solution that doesn’t cost educators more time, money, or stress.[13]

Significance

Educators have widely accepted the importance of using data to inform their treatment of students’ needs.[14][15] This is a good thing, as research touts the benefits of effective data use.[16][17][18][19] Unfortunately, educators’ widespread data use is not always a good thing. A significant portion – and some research claims most – of educators analyzing and using data are doing so incorrectly.[18][20][21][22][23][24] For example in two U.S. Department of Education studies conducted in districts known for strong data use, teachers achieved only 48% accuracy when making data inferences involving basic statistical concepts.[12][25] Thus educators are using data to inform decisions, but they do not always understand the data they are using. Since their data-misinformed decisions impact the students such decisions are meant to impact, this is a significant problem. Edtech products that present data to educators in an over-the-counter format – as opposed to simply “showing the data” and requiring educators to dig up resources to aid analyses – play an active role in improving educators’ data use.

Over-the-Counter Data Study

Diagram of over-the-counter data components
Infographic of OTCD study

Though numerous studies over the years have produced evidence on which the OTCD standards are based, one quantitative study in 2013 focused specifically on OTCD’s direct impact on data analysis accuracy (as opposed to merely determining which edtech aspects educators prefer). 211 educators of varied backgrounds at nine schools in six different California school districts participated in the Over-the-Counter Data’s Impact on Educators’ Data Analysis Accuracy study.[13] The study’s premise was to determine the precise impact on analysis accuracy when data system reporting environments made data “over-the-counter,” giving educators embedded supports like the kind over-the-counter medication provides for users in the form of labeling and supplemental documentation. Key findings were significant and hold implications for educators, educational technology and/or data system vendors, and anyone else involved in communicating data to educators:[26]

Relating to primary research questions

Relating to Secondary Research Questions

Over-the-Counter Data (OTCD) Standards

OTCD Standards involve embedding data analysis supports directly within reporting environments and adhering to best practices concerning design.[27] OTCD Standards were designed to be used by anyone communicating data to educators and to be reflected in the tool(s) through which the data is communicated (e.g., data report, data system, or other edtech product with a data component). Their purpose is to foster optimal educator (“user”) understanding, analysis, and use of the data being provided.

Mentions of OTCD

Organizations’ and publications’ mentions of OTCD include:

References

  1. ↑ Rankin, J. G. (2013, June 3). Featured article: What data reporting systems can learn from medicine labeling. EdSurge. Retrieved from https://www.edsurge.com/n/2013-06-03-opinion-what-data-reporting-systems-can-learn-from-medicine-labeling
  2. ↑ Rankin, J. (2013, October 24). Remedying educators’ data analysis errors with over-the-counter data. California Council on Teacher Education (CCTE) Conference. Poster presentation conducted from Kona Kai Resort, San Diego, CA.
  3. ↑ Rankin, J. G. (2013, May 2). Over-the-counter data is the next frontier for data in edtech. Edukwest. Retrieved from http://www.edukwest.com/over-the-counter-data-is-the-next-frontier-for-data-in-edtech/
  4. ↑ Rankin, J. (2011, December 1). Data analysis doesn’t have to be so hard. Ninetieth Annual California Educational Research Association (CERA) Conference Presentation conducted from Disney Convention Center, Anaheim, CA.
  5. ↑ DeWalt, D. A. (2010). Ensuring safe and effective use of medication and health care: perfecting the dismount. The Journal of the American Medical Association (JAMA), 304(23), 2641-2642. doi: 10.1001/jama.2010.1844
  6. ↑ Brown-Brumfield, D., & DeLeon, A. (2010). Adherence to a medication safety protocol: Current practice for labeling medications and solutions on the sterile field. Association of Operating Room Nurses. AORN Journal, 91(5), 610-610-7. doi:10.1016/j.aorn.2010.03.002
  7. ↑ Hampton, T. (2007). Groups urge warning label for medical devices containing toxic chemical. The Journal of the American Medical Association (JAMA), 298 (11), 1267. doi: 10.1001/jama.298.11.1267
  8. ↑ Qin, Y., Wu, M., Pan, X., Xiang, Q., Huang, J., Gu, Z., & ... Zhou, M. (2011, February 25). Reactions of Chinese adults to warning labels on cigarette packages: a survey in Jiangsu Province. BMC Public Health, 11(133). doi: 10.1186/1471-2458-11-133
  9. ↑ Rankin, J. G. (2013, October 25). Pushing edtech’s responsibility to communicate feedback effectively. Edtech Women. Retrieved from http://edtechwomen.com/blog/2013/10/25/pushing-edtechs-responsibility-to-communicate-feedback-effectively
  10. ↑ Rankin, J. (2013, May 7). Over-the-counter data: Improved analysis accuracy. Connect 2013: Canada’s Learning & Technology Conference. Presentation conducted from Scotiabank Convention Centre, Niagara Falls, Ontario, Canada.
  11. ↑ Kronstadt, J., Moiduddin, A., & Sellheim, W. (March, 2009). Consumer use of computerized applications to address health and health care needs: Prepared for U.S. Department of Health and Human Services, Office of the Secretary, Assistant Secretary for Planning and Evaluation. Bethesda, MD: NORC at the University of Chicago.
  12. 1 2 U.S. Department of Education Office of Planning, Evaluation and Policy Development (2009). Implementing data-informed decision making in schools: Teacher access, supports and use. United States Department of Education (ERIC Document Reproduction Service No. ED504191)
  13. 1 2 3 Rankin, J. G. (2013). Over-the-counter data’s impact on educators’ data analysis accuracy. ProQuest Dissertations and Theses, 3575082. Retrieved from http://pqdtopen.proquest.com/doc/1459258514.html?FMT=ABS
  14. ↑ Van der Meij, H. (2008). Designing for user cognition and affect in a manual. Should there be special support for the latter? Learning & Instruction, 18(1), 18-29.
  15. ↑ Hattie, J. (2010). Visibly learning from reports: The validity of score reports. Online Educational Research Journal. Retrieved from http://www.oerj.org/View?action=viewPaper&paper=6
  16. ↑ Wohlstetter, P., Datnow, A., & Park, V. (2008). Creating a system for data-driven decision-making: Applying the principal-agent framework. School Effectiveness and School Improvement, 19(3), 239–259.
  17. ↑ Lewis, D., Madison-Harris, R., Muoneke, A., & Times, C. (2010). Using data to guide instruction and improve student learning. SEDL Letter, 22(2), 10–12.
  18. 1 2 SAS Institute. (2013). Best practices in information management, reporting and analytics for education. Retrieved from https://fs24.formsite.com/edweek/form15/secure_index.html
  19. ↑ Stansbury, M. (2013, July). Nine templates to help educators leverage school data: New industry collaborative says using data effectively can help close education gaps. eSchool News. Retrieved from http://www.eschoolnews.com/2013/01/07/nine-templates-to-help-educators-leverage-school-data/?ast=104&astc=9990
  20. ↑ Data Quality Campaign (2009). The next step: Using longitudinal data systems to improve student success. Retrieved from http://www.dataqualitycampaign.org/find-resources/the-next-step/
  21. ↑ Wayman, J. C., Cho, V., & Shaw, S. M. (2009, December). First-year results from an efficacy study of the Acuity data system. Paper presented at the Twenty-fourth Annual Texas Assessment Conference, Austin, TX.
  22. ↑ Underwood, J. S., Zapata-Rivera, D., & VanWinkle, W. (2010). An evidence-centered approach to using assessment data for policymakers (ETS Research Rep. No. RR-10-03). Princeton, NJ: ETS.
  23. ↑ Wayman, J. C., Snodgrass Rangel, V. W., Jimerson, J. B., & Cho, V. (2010). Improving data use in NISD: Becoming a data-informed district. Austin, TX: The University of Texas at Austin.
  24. ↑ VanWinkle, W., Vezzu, M., & Zapata-Rivera, D. (2011). Question-based reports for policymakers (ETS Research Memorandum No. RM-11-16). Princeton, NJ: ETS.
  25. ↑ U.S. Department of Education Office of Planning, Evaluation and Policy Development (2011). Teachers' ability to use data to inform instruction: Challenges and supports. United States Department of Education (ERIC Document Reproduction Service No. ED516494)
  26. ↑ Hansen, L., & Johnson, M. (2013, July 24) Data-Informed Decision Making: It Takes a City. Learning Forward (formerly National Staff Development Council) 2013 Summer Conference. Presentation conducted from Minneapolis Hilton, Minneapolis, MN
  27. ↑ Rankin, J. (2013, March 28). How data systems & reports can either fight or propagate the data analysis error epidemic. 2013 Technology Information Center for Administrative Leadership (TICAL) School Leadership Summit. Retrieved from http://admin20.org/forum/topics/how-data-systems-reports-can-either-fight-or-propagate-the-data
This article is issued from Wikipedia - version of the Saturday, January 03, 2015. The text is available under the Creative Commons Attribution/Share Alike but additional terms may apply for the media files.