Public Perception of Helical Rim Deformities and Their Correction with Ear Molding

Original Article

Public Perception of Helical Rim Deformities and Their Correction With Ear Molding

Viren Patel, BS, Daniel M. Mazzaferro, MD, Jordan W. Swanson, MD, Jesse A. Taylor, MD, and Scott P. Bartlett, MD

Background: Among congenital ear deformities, helical rim defor- mities are most common. Non-surgical ear molding has emerged as an effective option to treat helical rim deformities and could reduce the need for surgery later in a child’s life. Despite this, there has never been a study examining how the general public rates correc- tions after ear molding.

Methods: A retrospective chart review was conducted of all consecutive patients with helical rim deformities treated with InfantEar TM Molding System. Amazon Mechanical Turk (MTurk) was used to survey blinded respondents using photographs of patients’ ears to determine the degree of normalcy on a scale of 1 (not normal) to 10 (completely normal) before and after molding. Ratings of ears were compared using a paired t test and percent improvement from baseline.

Results: A total of 59 ears met criteria for evaluation by 497 MTurk respondents. Average age of patients at the time of treatment was 34.2 ± 16.8 days and mean therapy duration was 31.3 ± 13.1 days. A paired t test analysis found that MTurk respondents identified significant improvement in 91.5% of ears (n ¼ 54) (P < 0.05). The percent improvement in observed helical rim change was on average 82.7 ± 61.1%.

Conclusion: Ear molding is an effective and safe way to address helical rim deformities early in the neonatal period. The general public is able to recognize ears with rim deformities as abnormal, and appreciate the difference in architecture after molding. Crowdsourcing technology offers an opportunity to measure laypeople’s perceptions regarding outcomes after ear molding.

---

From the Division of Plastic Surgery, The Children’s Hospital of Philadelphia, Philadelphia, PA.

Received October 14, 2019.

Accepted for publication January 23, 2020.

Address correspondence and reprint requests to Scott P. Bartlett, MD, Professor of Surgery, The Perelman School of Medicine, The Children’s Hospital of Philadelphia, Abrahamson Pediatric Cancer Center, 1st Floor, 3615 Civic Center Blvd., Philadelphia, PA 19104; E-mail: bartletts@email.chop.edu

SB is the founder and shareholder of TalexMedical (Villanova, PA). The Children’s Hospital of Philadelphia has a patent pending for the intel- lectual property associated with the InfantEar system. No financial support or benefits from TalexMedical was provided for any author in the execution of this study and preparation of this manuscript.

The authors report no conflicts of interest.

Supplemental digital contents are available for this article. Direct URL citations appear in the printed text and are provided in the HTML and PDF versions of this article on the journal’s Web site (www.jcraniofa- cialsurgery.com).

Copyright # 2020 by Mutaz B. Habal, MD ISSN: 1049-2275

DOI: 10.1097/SCS.0000000000006400

Key Words: Congenital ear anomaly, craniofacial, craniofacial

disfigurement, ear molding, earwell, helical rim deformity, infantear, otoplasty

(J Craniofac Surg 2020;31: 741–745)

Congenital auricular deformities are common and often noticed first by parents in the neonatal period. Deformations are defined as anomalies in which the skin and cartilage are present, but there is some distortion of normal ear architecture (as opposed to malformations, in which a problem during embryologic develop- ment leads to missing skin, cartilage, or both). While the true incidence of congenital ear deformities is unclear, multiple studies have reported that over half of neonates have some degree of ear abnormality. This may be an underestimate, as less severe defor- mations often go unrecognized by parents and providers.^1,2 More- over, reported rates of self-correction of these abnormalities vary greatly, with estimates ranging from 30% to 80%.^1–4 Notably, even in studies that found higher rates of self-correction, the authors emphasized the inability to predict which deformities will persist.^1,5 The malleability of neonatal cartilage, under the influence of maternal estrogen (returning to normal fetal levels at 6 weeks post- birth) has promoted the use of a variety of splinting methods to correct these deformities.^6–9 Matsuo² was one of the first of many to popularize early splinting showing excellent corrections of these abnormalities.^2,3,9–12 Most studies have emphasized the importance of early intervention when maternal estrogen levels are still high.^2,3,10 However, there has been reports of similar improvements with molding later in a child’s life.^9,13

While there have been many studies looking at the effectiveness of splinting for congenital ear deformities, almost all have used evaluations by plastic surgeons to rate the success of treatment.^{2,3,9– 11} No study has looked at the lay-public’s perception of the degree of deformation nor the improvement after treatment. The impor- tance of this perspective is underscored by the fact that that surgeons’ opinions do not always align with laypeople’s perspec- tives, as physicians have been shown to be more critical of facial discrepancies than the general public.^14,15 Woo¹² was unique in that parents’ impressions of corrections were included. However, par- ents’ perspectives of corrections after molding are likely not representative of the population as a whole.¹²

Part of the reason the general public’s perspective is missing from the literature is due the challenges of creating a sample large enough to be representative of the lay-public. However, with the emergence of Amazon’s Mechanical Turk (MTurk), a crowdsour- cing platform, conducting a large survey has become fast and relatively inexpensive.¹⁶ Samples generated from MTurk are more diverse and representative of the general population when com- pared to traditional sampling methods.^16–19 Additionally, samples from MTurk have already began to appear in the literature to evaluate the results of other plastic surgeries.^20–23 For these rea- sons, Amazon’s MTurk meets, if not exceeds, the standards set by traditional convenience sampling.

METHODS

Technique

While there are many commercially available devices, we prefer the InfantEar Neonatal Molding System. The device incor- porates a Velcro base and soft silicone conformers which are all contoured to patients’ ears with a silicone gel. The use of the silicone gel is what differentiates InfantEar from other devices on the market, as it allows for more flexibility when fitting the patients’ ears and customizes the construct to the individual defor- mity. The other devices are often rigid and require external frames that are easily dislodged and may fit poorly. Furthermore, the InfantEar device is generally easier to apply than other devices on the market and is currently used by both specialists and general pediatricians.

Patients are evaluated by one of three surgeons to determine if molding would be efficacious in improving their deformities. Generally, this decision is based on physical exam at patients’ initial visit and their age at presentation. While there is no age cut- off used at our institution, older patients tend to have less pliable tissue and therefore, are not offered molding. When a patient is deemed to be a suitable candidate for molding, he hair behind the auricle is shaved and the skin is cleansed with a soap wipe. The precut Velcro base is applied, and courses from the skin anterior to the base of the helix posteriorly around the ear to the lobule (Fig. 1A). If necessary, a helical rim guide (silicone-coated wire) can be placed, but is not necessary (Fig. 1B). Helical rim con- formers are placed along the ear in locations that would result in the most normal ear architecture (Fig. 1C). The rim conformers are fixed to a Velcro base piece (Fig. 1D). The silicone gel flows into all voids, filling the spaces on the anterior and posterior surfaces of the ear and between the Velcro base and the helical rim conformers, fixing and the device in place (Fig. 1E). A plastic cover with Velcro is attached to the device after the gel solidifies (Fig. 1F).

After application, little care is required by the parents. They are instructed to keep the device dry as moisture may cause skin irritation or device loosening. Generally, the device will stay on the infant’s ear for 2 to 3 weeks, but if the parents notice it starting to separate, they are instructed to use cyanoacrylate adhesive to fix the Velcro base back to the skin. All patients are seen in the office 2 to 3 weeks after device application to check for any superficial skin reactions. Depending on the degree of correction and age at initial application, a new device may be applied on one or more occasions.

Patient Selection

An IRB approved retrospective chart review was conducted to  identify  all  consecutive  infants  treated  with  InfantEar device from January 2013 to January 2019. Subjects with helical rim deformities, defined as pleating or abnormal folding of the auricular rim were identified. Subjects were included in the study if they were diagnosed with an isolated helical rim deformity and had successfully completed treatment with before and after molding photos available in the electronic medical record (EMR). Demographic and treatment information were abstracted from the EMR.

Mild skin irritation surrounding the ear is not an uncommon finding at the time of device removal. Generally, this is thought to be a mild reaction to the adhesive or a moisture rash and resolves completely within a couple days of device removal. All parents are instructed to return to the clinic 6 weeks after device removal to ensure that there has been no recurrence of the deformity and that the patient’s skin irritation resolved and to return sooner if recur- rence is noted before that time. Despite this, most parents opt not to travel back to our institution, as we serve a large catchment area and parents travel long distances to visit our clinic. Therefore, many of the pictures available to us were taken immediately after device removal and had some element of skin irritation. Telephone and E- mail encounter data was reviewed for each patient to determine if patients had any issues after device removal. Photos of ears with significant skin irritation or crusting were chosen to be excluded from the MTurk portion of the study, as the superficial skin appearance could influence how MTurk participants rated the normalcy of the ear shape. Despite being excluded from submission to MTurk, these patients were included in discussions about com- plications as to avoid underreporting complication rates. Similarly, poor resolution photos that detracted from accurately visualizing the ear contours were not submitted to MTurk.

MTurk Design

MTurk was used to identify and compensate participants who completed a survey on Qualtrics (Qualtrics, Provo, Utah). In order to limit insufficient effort responding (IER), as described by Huang, several measures were included to ensure the quality of the data collected from respondents.²⁴ Survey respondents were required to have a minimum of a 95% approval rating and be living in the United States. Before starting the survey, participants were asked a set of demographic questions in addition to 2 control questions about the current president and year. If participants incorrectly answered either of the 2 control questions, their responses were excluded from analysis. If participants met all criteria, they were randomized into 1 of 2 groups that contained half of the subject’s ears. This was done to prevent fatigue and inattention that would occur if raters were asked to assess too many ears. Raters were shown pictures of subject’s ears in random order, blinded to which were before after treatment, and asked to rate the normalcy on a scale of 1 to 10.

As additional means to control for IER, three photos of normal ears, chosen by the senior plastic surgeon, were included as controls in the survey. Each MTurk participant’s rating of the normal ears were compared to the average of their ratings for all of the subjects’ ears. If MTurk participants rated the normal ears lower than the average of their other ratings, their responses were excluded from analysis. Furthermore, members of the research team attempted to click through the survey as quickly as possible, to determine the fastest time that the survey could be completed. Any rater who finished the survey quicker than the average time of the team, was also excluded, as these raters were likely producing poor quality data.²⁵

Statistical Analysis

Stata, Version 13 (StataCorp, College Station, TX) was used for data analysis and standard descriptive statistics were conducted. Ratings of ears before and after treatment were compared using paired t tests, with significance set at a P value under 0.05. The percent improvement above baseline was also calculated for each MTurk participant. Using Kruskal– Wallis, the percent improve- ment above baseline was used to compare how raters of different demographics rated corrections.

RESULTS

Molding Patients

A total of 55 subjects (29 males, 26 females) completed splinting treatment for a helical rim deformity between January 2013 to January 2019, for a total of 85 ears (Supplemental Digital Content, Table 1, http://links.lww.com/SCS/B296). Average age at treatment initiation was 34.2 days (range 6– 76 days) and average treatment length was 31 days (range 7–72 days). The majority of subjects (53%) required 2 applications of the ear molding before achieving appropriate corrections.

While 14 patients (25%) developed complications during mold- ing treatment, all were mild and resolved completely. Three patients (5.4%) were noted to have regression after treatment, while eleven patients (19.6%) were found to have local skin irritation or small pressure ulcers (Fig. 2). All complications resolved after the mold was removed without any additional intervention, with the excep- tion of 1 subject who required topical steroids. There was no significant association between age at initial application, length of treatment, or number of molding applications relative to the development of complications (Supplemental Digital Content, Table 2, http://links.lww.com/SCS/B296).

After review with the senior plastic surgeon, photos of 16 ears were excluded from evaluation by MTurk raters due to poor picture quality. Another 10 ears were excluded from MTurk analysis due superficial skin reaction, with 1 ear having ulcers, 6 having local erythematous irritation and 3 with crusting. Therefore, a total of 59 ears were evaluated by MTurk participants.

Mturk Raters

A total of 59 ears met criteria and were evaluated by MTurk participants. A total of 602 MTurk respondents completed the survey, with 302 rating one set of ears (Group 1) and the other 300 randomized to the other set (Group 2). Six respondents were excluded for incorrectly answering one of the control questions. Another nine were excluded for completing the survey under the minimum time, while 90 were excluded for failing to rate the normal control ears above their individualized threshold. This left a total of 497 MTurk respondents for analysis (244 respondents in Group 1 and 253 respondents in Group 2). On average, respondents took 8.3 minutes to complete the survey. The majority of respondents were female (65%), married (47%) and aged between 20 and 49 years old (61%) (Supplemental Digital Content, Table 3, http://links.lww.com/SCS/B296).

Improvement in Ear Normalcy

Out of all the ears rated by MTurk respondents, 54 corrected ears (91.5%) were found to have significantly higher ratings when compared to ratings of pre-treatment ears. Figures 3 to 5 show 3 examples of the 54 ears that had significant improvement when comparing before and after treatment photos, which ranged from a 79% to 184% improvement.

MTurk respondents who were randomized to Group 1 rated post- treatment ears, on average, 99% higher than pre-treatment ears. Group 2 MTurk respondents rated post-treatment ears, on average, 66% higher than pre-treatment ears. Percent improvement from baseline scores of pre-treated ears were also used to compare how different demographic groups rated ear corrections. In Group 2, no differences were found between how demographic sub-groups rated corrected ears versus non- treated ears (P > 0.05) In Group 1, no differences were found between any demographic sub-groups except between MTurk respondents with different education levels (P < 0.05).

DISCUSSION

It is established that molding is an effective treatment for congenital auricular deformities. However, until this study, the public’s per- ception of ears corrected with molding had not been examined. Using Amazon’s MTurk, we have generated a sample that is able to act as a proxy for how the lay-public would evaluate corrections after ear molding with InfantEar. We have shown that ear molding is very effective in treating congenital helical rim deformities, with 91.5% of ears entered showing significant improvement after treatment. This success rate is similar to that previously reported in studies which used plastic surgeons to evaluate corrections.^3,9,12 The results of this study show that laypersons are able to differ- entiate between normal and abnormal ear architecture, even in subtle helical rim abnormalities. Correction of these deformities has a significant impact on, not only children’s appearance, but also their psychological well-being, as correction of auricular abnormalities is associated with improved self-esteem.^26–29 Moreover, children with craniofacial disfigurements have been found to have higher rates of anxiety and behavioral problems, including social withdrawal.^30–32 Many of these psychological differences stem from emotional disturbance sustained due to teasing from other children.^26,33–35 Horlock¹⁴ showed that over 60% of children seeking ear reconstruc- tion were motivated by teasing or wanting to be accepted by peer groups, and over 85% reported improvement in these motivators after correction. Considering that peer’s perception of patient’s ears was a major motivator for pursuing correction and this same perception contributed to the improvements in psychosocial well-being, it is important to show that the lay-public views these corrections positively, and our study is the first to do so.

There continues to be questions about the ideal timing of mold application, with some studies showing improved results with early molding and others suggesting treatment later in life can also result in significant improvement.^2,3,9,10,13 In our population, over 75% of patients initiated molding within the first 50 days of life. While some subjects in this study had positive results, even at 76 days of life, it was not common for our institution to offer molding for patients at that age. Therefore, further studies are needed to assess if molding at ages after the first 2 months of life are effective at treating these deformities. Regardless, the results from this study underscore the importance of early detection and referral by general pediatricians. At our center, patients often had to undergo a waiting period between initial evalua- tion and initial mold application, for authorization by their insurances. Therefore, early referral is paramount in beginning treatment early in the infantile period and achieving appropriate results.

When looking at the five ears that were not rated as significantly improved, a number of observations can be made. For three of the ears, the deformity was very mild, with small pleating of the helical rim. When reviewing the images with the senior plastic surgeon, there is improvement after treatment, but the difference is small and unlikely to be noticed by the untrained eye. In another ear, there is improvement of the helical rim deformity, but after treatment, the fold is still not fully corrected (Fig. 6). For the final ear in this group, there is noticeable improvement of shape of the rim, but there is some slight erythema noted on the post-treatment image that could have confounded how respondents rated this ear.

This study was only possible due to MTurk’s ability to gather raters quickly with relatively low cost. In addition, MTurk respondents are blinded, randomly presented with a series of patient photos, and are not exposed to evaluator bias (contrary to evaluation by the treating physician). However, the data cannot be assessed without considering the inherent limitations of utilizing MTurk. Insufficient error in responding is a major concern for any data collected through MTurk, as participants are incentivized to complete as many surveys as possible to maximize payout. However, studies comparing responses from MTurk to traditional sampling methods have found no difference in the quality of the data collected, as traditional sampling also results in subjects not responding in full faith.^36–40

Regardless, the study design includes multiple methods to limit the impact of IER on the dataset. First, access to the survey is limited to MTurk raters who have HIT (Human Intelligence Task) approval ratings over 95%, which would limit raters based on the quality of responses in prior surveys. While this assumes that quality of responses in the past correlate with future data quality, there is evidence to suggest MTurk raters with HITs above 95% score better on tests of attentiveness than users with HIT under 95%.³⁹ MTurk subjects were also required to correctly answer the 2 control questions to access the survey as an attention check, which has been shown to improve data reliability.^25,41 Furthermore, each MTurk subject was only shown half of the ears, in an effort to limit inattention as a result of having to rate too many photos. While having 2 separate groups makes it challenging to compare demo- graphics, the authors feel that the large sample size allows for accurate normalcy ratings for each ear.

There were also measures taken after data collection to ensure the validity of the data used for analysis. Screening out raters who finished the survey too quickly is a well-established means to control for IER.²⁵ Additionally, raters were screened using the normal ears as control images. Some studies have suggested that removing seem- ingly inconsistent responses affects the validity of the data while others suggest that this practice must be employed to screen respon- dents who are obviously not paying attention to questions.^17,42,43 By using the normal ears as a control, the authors feel that the removed respondents were not giving full attention to questions and their exclusion improved the validity of the data collected.

There are a number of limitations to this study that could have impacted the results. Since inclusion criteria required patients to have completed treatment, patients who did not complete treatment may have had complications or poor results, which would not be represented by this dataset. While this may be true for a small subset of subjects, there were a far greater number of subjects who had positive results and chose not to follow-up, based on telephone encounter information in the EMR. Therefore, this inclusion criteria likely led to less favorable results, than if patients lost to follow-up had been included. Moreover, these subjects were chosen to be excluded because there were no post-treatment photos that could have been used for evaluation by MTurk raters.

Another limitation stems from the decision to only include helical rim deformities. This was done because it was the most common deformity in our population. While we have found an improvement in laypersons’ perspectives in helical rim deformities, this cannot be generalized to other types of deformities. The authors hypothesize a similar or greater improvement would exist for other deformities, including cupped ears, Stahl’s ear, and cryptotia, but additional studies are needed to test this hypothesis.

REFERENCES

1. Zhao H, Lin G, Seong YH, et al. Anthropometric research of congenital auricular deformities for newborns. J Matern Fetal Neonatal Med 2019;32:1176–1183
2. Matsuo K, Hayashi R, Kiyono M, et al. Nonsurgical correction of congenital auricular deformities. Clin Plast Surg 1990;17:383–395
3. Byrd HS, Langevin CJ, Ghidoni LA. Ear molding in newborn infants with auricular deformities. Plast Reconstr Surg 2010;126:1191–1200
4. Smith W, Toye J, Reid A, et al. Nonsurgical correction of congenital ear abnormalities in the newborn: Case series. Paediatr Child Health 2005;10:327–331
5. Kamiishi H. Evaluation of a simple prosthesis for correction of cryptotia. Keisei Geka Plast Reconstruct Surg 1976;19:7–12
6. Kenny FM, Angsusingha K, Stinson D, et al. Unconjugated estrogens in the perinatal period. Pediatr Res 1973;7:826–831
7. Hardingham TE, Muir H. Hyaluronic acid in cartilage and proteoglycan aggregation. Biochem J 1974;139:565–581
8. Schiff M, Burn HF. The effect of intravenous estrogens on ground substance. Arch Otolaryngol 1961;73:43–51
9. Chang CS, Bartlett SP. A Simplified nonsurgical method for the correction of neonatal deformational auricular anomalies. Clin Pediatr (Phila) 2017;56:132–139
10. Tan ST, Shibu M, Gault DT. A splint for correction of congenital ear deformities. Br J Plast Surg 1994;47:575–578
11. Tian Y, Wang F, Yu JJ, et al. Congenital ear deformity screening and non-invasive correction effect analysis. Lin Chung Er Bi Yan Hou Tou Jing Wai Ke Za Zhi 2019;33:259–261
12. Woo T, Kim YS, Roh TS, et al. Correction of congenital auricular deformities using the ear-molding technique. Arch Plast Surg 2016;43:512–517
13. Yotsuyanagi T, Yokoi K, Urushidate S, et al. Nonsurgical correction of congenital auricular deformities in children older than early neonates. Plast Reconstr Surg 1998;101:907–914
14. Horlock N, Vo¨gelin E, Bradbury ET, et al. Psychosocial outcome of patients after ear reconstruction: a retrospective study of 62 patients. Ann Plast Surg 2005;54:517–524
15. Fabre´ M, Mossaz C, Christou P, et al. Orthodontists’ and laypersons’ aesthetic assessment of Class III subjects referred for orthognathic surgery. Eur J Orthod 2009;31:443–448
16. Mason W, Suri S. Conducting behavioral research on Amazon’s Mechanical Turk. Behav Res Methods 2012;44:1–23
17. Berinsky AJ, Huber GA, Lenz GS. Evaluating online labor markets for experimental research: Amazon.com’s mechanical turk. Political Anal 2012;20:351–368
18. Wolfson S, Bartkus JR. An assessment of experiments run on Amazon’s Mechanical Turk. Mustang J Bus Ethics 2013;5:119–139
19. Buhrmester M, Kwang T, Gosling SD. Amazon’s mechanical turk: a new source of inexpensive, yet high-quality, data? Perspect Psychol Sci 2011;6:3–5
20. Tse RW, Oh E, Gruss JS, et al. Crowdsourcing as a novel method to evaluate aesthetic outcomes of treatment for unilateral cleft lip. Plast Reconstr Surg 2016;138:864–874
21. Wu C, Scott Hultman C, Diegidio P, et al. What do our patients truly want? Conjoint analysis of an aesthetic plastic surgery practice using Internet crowdsourcing. Aesthet Surg J 2016;37:105–118
22. Naran S, Wes AM, Mazzaferro DM, et al. More than meets the eye: the effect of intercanthal distance on perception of beauty and personality. J Craniofac Surg 2018;29:40–44
23. Mazzaferro DM, Wes AM, Naran S, et al. Orthognathic surgery has a significant effect on perceived personality traits and emotional expressions. Plast Reconstr Surg 2017;140:971–981
24. Huang JL, Liu M, Bowling N. Insufficient effort responding: examining an insidious confound in survey data. J Appl Psychol 2015;100:828–845
25. Wood D, Harms P, Lowman G, et al. Response speed and response consistency as mutually validating indicators of data quality in online samples. Soc Psychol Personal Sci 2017;8:454–464
26. Bradbury ET, Hewison J, Timmons MJ. Psychological and social outcome of prominent ear correction in children. Br J Plast Surg 1992;45:97–100
27. Arndt EM, Lefebvre A, Travis F, et al. Fact and fantasy: psychosocial consequences of facial surgery in 24 Down syndrome children. Br J Plast Surg 1986;39:498–504
28. Pertschuk MJ, Whitaker LA. Social and psychological effects of cranio- facial deformity and surgical reconstruction. Clin Plast Surg 1982;9:297–306
29. Pillemer FG, Cook KV. The psychosocial adjustment of pediatric craniofacial patients after surgery. Cleft Lip Palate J 1989;26:201–208
30. Kapp Simon KA, Simon DJ, Kristovich S. Self-perception, social skills, adjustment, and inhibition in young adolescents with craniofacial anom- alies. Cleft Palate Craniofac J 1992;29:352–356
31. Broder H, Strauss RP. Self-concept of early primary school age children with visible or invisible defects. Cleft Palate J 1989;26:114–117
32. Speltz ML, Morton K, Goodell EW, et al. Psychological functioning of children with craniofacial anomalies and their mothers: follow-up from late infancy to school entry. Cleft Palate Craniofac J 1993;30:482–489
33. Endriga MC, Kapp Simon KA. Psychological issues in craniofacial care: state of the art. Cleft Palate Craniofac J 1999;36:3–11
34. Macgregor FC. Ear deformities: social and psychological implications. Clin Plast Surg 1978;5:347–350
35. BradburyET. Psychological approaches to children and adolescents with disfigurement. ACPP Rev Newsletter 1993;15:1–6
36. Crump MJ, McDonnell JV, Gureckis TM. Evaluating Amazon’s Mechanical Turk as a Tool for Experimental Behavioral Research. PLoS One 2013;8:e57410
37. Necka EA, Cacioppo S, Norman GJ, et al. Measuring the prevalence of problematic respondent behaviors among MTurk, Campus, and community participants. PLoS One 2016;11:e0157732
38. Klein RA, Ratliff KA, Vianello M, et al. Investigating variation in replicability: a ‘‘many labs’’ replication project. Soc Psychol (Gott) 2014;45:142–152
39. Horton JJ, Rand DG, Zeckhauser RJ. The online laboratory: conducting experiments in a real labor market. Exp Econ 2011;14:399–425
40. Paolacci G, Chandler J, Ipeirotis PG. Running experiments on Amazon Mechanical Turk. Judgm Decis Mak 2010;5:411–419
41. Oppenheimer DM, Meyvis T, Davidenko N. Instructional manipulation checks: detecting satisficing to increase statistical power. J Exp Soc Psychol 2009;45:867–872
42. Berninsky AJ, Margolis MW, Sances MW. Separating the shirkers from the workers? Making sure respondents pay attention on self- admisinstered surveys. Am J Political Sci 2014;58:739–753
43. Berninsky AJ, Margolis MW, Sances MW. Can we turn shirkers into workers? J Exp Soc Pscyhol 2016;66:20–28