|Year : 2018 | Volume
| Issue : 2 | Page : 165-171
Test–Retest Reliability of the Short-Form Survivor Unmet Needs Survey
Karen Taylor1, Max Bulsara2, Leanne Monterosso3
1 School of Nursing and Midwifery, University of Notre Dame Australia; Western Australia Cancer and Palliative Care Network, Perth, Australia
2 Institute for Health Research, University of Notre Dame Australia, Fremantle, Australia
3 School of Nursing and Midwifery, University of Notre Dame Australia, Perth; St John of God Murdoch Hospital, Murdoch; School of Nursing, Edith Cowan University Joondalup, Western Australia, Australia
|Date of Submission||22-Dec-2017|
|Date of Acceptance||22-Jan-2018|
|Date of Web Publication||9-Mar-2018|
Western Australia Cancer and Palliative Care Network, Perth, WA
Source of Support: None, Conflict of Interest: None
Objective: Reliable and valid needs assessment measures are important assessment tools in cancer survivorship care. A new 30-item short-form version of the Survivor Unmet Needs Survey (SF-SUNS) was developed and validated with cancer survivors, including hematology cancer survivors; however, test–retest reliability has not been established. The objective of this study was to assess the test–retest reliability of the SF-SUNS with a cohort of lymphoma survivors (n = 40). Methods: Test–retest reliability of the SF-SUNS was conducted at two time points: baseline (time 1) and 5 days later (time 2). Test–retest data were collected from lymphoma cancer survivors (n = 40) in a large tertiary cancer center in Western Australia. Intraclass correlation analyses compared data at time 1 (baseline) and time 2 (5 days later). Cronbach's alpha analyses were performed to assess the internal consistency at both time points. Results: The majority (23/30, 77%) of items achieved test–retest reliability scores 0.45–0.74 (fair to good). A high degree of overall internal consistency was demonstrated (time 1 = 0.92, time 2 = 0.95), with scores 0.65–0.94 across subscales for both time points. Conclusions: Mixed test–retest reliability of the SF-SUNS was established. Our results indicate the SF-SUNS is responsive to the changing needs of lymphoma cancer survivors. Routine use of cancer survivorship specific needs-based assessments is required in oncology care today. Nurses are well placed to administer these assessments and provide tailored information and resources. Further assessment of test–retest reliability in hematology and other cancer cohorts is warranted.
Keywords: Cancer survivorship, internal consistency, lymphoma, short-form Survivor Unmet Needs Survey, test–retest reliability
|How to cite this article:|
Taylor K, Bulsara M, Monterosso L. Test–Retest Reliability of the Short-Form Survivor Unmet Needs Survey. Asia Pac J Oncol Nurs 2018;5:165-71
|How to cite this URL:|
Taylor K, Bulsara M, Monterosso L. Test–Retest Reliability of the Short-Form Survivor Unmet Needs Survey. Asia Pac J Oncol Nurs [serial online] 2018 [cited 2019 Jan 19];5:165-71. Available from: http://www.apjon.org/text.asp?2018/5/2/165/227088
| Introduction|| |
Lymphoma blood cancers are malignant T or B cell lymphocytes in the lymphatic system and are categorized under two main types: non-Hodgkin lymphoma (NHL) and Hodgkin Lymphoma (HL). NHL represents approximately 88% of all lymphomas, while HL is predominately diagnosed in the adolescent and young adult population. Combined, they represent the sixth most common cancer diagnosis worldwide. Consistent with worldwide trends, the incidence of lymphoma in Australia is increasing, and with a projected diagnosis of 6232 cases in 2017, this equates to 4.6% of all cancer cases. An estimated mortality rate of 1481 equates to 3.1% of all deaths from cancer in 2017. Projected figures for 2017 in the USA have a similar projected incidence of lymphoma of 4.8% and mortality of 3.6%. Treatment for lymphoma generally comprises high-dose chemotherapy and/or targeted immunotherapy agents and may include radiotherapy and hematopoietic stem cell transplants. These treatments have resulted in an improvement to overall survival of approximately 76% at 5 years compared with 52% at 5 years in the 1980s. Notwithstanding the positive impact treatment has had on survival rates, the consequences of disease and treatment continue long after treatment completion. Long-term and late effects may produce ongoing unmet needs such as fear of recurrence, fatigue, poor nutrition, exercise, fertility, relationship, financial, employment, and insurance issues.
To provide optimal supportive cancer care to lymphoma survivors, the identification of patients' perceived concerns and level of support needed is required. This is especially important for younger patients (18–45 years of age) where the expectation of long-term remission can raise additional concerns and unmet needs. Receiving relevant information and practical support soon after treatment ends, especially resources related to healthy lifestyle behaviors,,,,, can help mitigate the impact of disease and treatment and lead to fewer unmet needs further along the survivorship continuum., A qualitative study with lymphoma cancer survivors (n = 17) undertaken in Western Australia  reported unmet informational and practical needs as participants transitioned from treatment to the survivorship phase. The findings suggested tailored post-treatment support and interventions are fundamental components of excellent survivorship care.
The measures used to assess unmet needs are equally important. Generic cancer measures which comprise items related to diagnosis and treatment are often not specific enough for the survivorship phase. Comprehensive, relevant, reliable, and validated needs assessment measures that are survivor-specific are essential to capture unmet needs that become evident when treatment ends. These measures can guide health professionals in providing individualized information, support, and resources., Two recent systematic reviews , revealed that needs assessment tools are varied and may not capture all the possible unmet needs patients may have. The reviews likewise found validity and reliability evidence limited. The Survivor Unmet Needs Survey (SUNS) was identified as a measure that had strong psychometric properties and was developed and psychometrically tested with a large cross-sectional sample of cancer survivors (n = 550) including a small cohort of hematology cancer participants (n = 31, 5.6%). Campbell et al. confirmed a high overall internal consistency of items for their study with an overall Cronbach's alpha of 0.99. The authors also reported high test–retest reliability although the results were not published. Internal consistency of the SUNS was further tested in two studies of hematological cancer survivor cohorts. A cross-sectional study with 529 hematological cancer survivors  demonstrated overall Cronbach's alpha values >0.9, and a weighted Kappa coefficient score of >0.6 for test–retest reliability; acceptability was reported for 40/89 (45%) items. Qualitative data from 17 semi-structured interviews indicated that the SUNS was considered relevant by this cohort of hematological cancer survivors. A cross-sectional study of hematological cancer survivors from Australia and Canada (n = 437) reported similar levels of unmet needs across the two cohorts using the SUNS, with fatigue (n = 76, 17%) and financial concerns (n = 39, 9%) rated as high unmet needs. Despite the clinical utility of the original SUNS, it was considered potentially burdensome for use in the clinical setting given the large number of items (n = 89). In 2014, the 30-item short-form-SUNS (SF-SUNS) was developed and validated with a mixed sample of cancer survivors (n = 1589), including hematological cancer survivors (n = 84, 5%). Construct validity and intraclass correlation coefficients (ICCs) of the SF. were similar to those of the original SUNS. Cronbach's alpha scores for the final four domains were ≥0.85, and ICCs for the three domains from the original SUNS (financial concerns, information, and access and continuity of care) and the SF-SUNS were high (<0.9). Discriminant validity demonstrated the SF-SUNS ability to discriminate between individuals who had recently received treatment and those who had not. The authors recommended further testing of the SF-SUNS for test–retest reliability. The 30-item SF-SUNS was therefore judged to be more practical and likely to be completed by participants in our larger study, particularly as the SF-SUNS was one of four instruments to be administered to participants in a pilot randomized trial to measure the effect of a nurse-led survivorship model of care.
For researchers and clinicians to develop targeted follow-up support for cancer cohorts underrepresented in survivorship literature, such as lymphoma, cohort-specific studies in the early survivorship phase are required. Therefore, this study recruited only those with a lymphoma diagnosis who had completed treatment. Discerning the issues and concerns of this group requires survivor-specific measures that are psychometrically sound and fully tested. The SF-SUNS has been used within the clinical setting; however, since test–retest reliability of the SF-SUNS had not been established, the aim of the present study was to establish test–retest reliability of the SFSUNS to add to the psychometric data available in the published literature on this instrument.
| Methods|| |
Test–retest reliability of the SF-SUNS was conducted at two time points: baseline (time 1) and 5 days later (time 2). This time frame was chosen to reduce recall bias and change in the level of unmet needs. Ethical approval to conduct the study was obtained from the human research ethics committee of the study site (2015-020) and university (015007F).
Population and setting
A convenience sample of 40 lymphoma cancer patients who were 3 months' posttreatment completion were recruited from the hematology department of a large tertiary hospital in Western Australia. Inclusion criteria were pathologically confirmed new diagnosis of NHL or HL; completedfirst-line curative intent chemotherapy or second-line curative intent autologous stem cell transplant within the previous 3 months; no radiological evidence of lymphoma posttreatment (on positron emission tomography [PET] scan); able to understand and read English; and over 18 years of age. Participants were excluded if they had not been treated with chemotherapy; had received further treatment at another hospital (as experiences or interventions may have introduced bias); or were cognitively impaired or experiencing an acute mental health condition that prohibited the provision of informed consent.
The sample size calculation was derived from Walter et al. and used a fixed alpha of 0.05 from two observations with reliability values of R0 = 0.6 (acceptable) and R1 = 0.8 (expected), indicating a minimum sample size of n = 39.
Short-form Survivor Unmet Needs Survey
The SF-SUNS assesses unmet needs across four domains: information needs (3 items); work and financial needs (8 items); access and continuity of care needs (6 items); and coping, sharing, and emotional needs (13 items). Patient self-reported concerns and the level of support required are measured using a Likert-type scale: 0 – no unmet need, 1 – low unmet need, 2 – moderate unmet need, 3 – high unmet need, and 4 – very high unmet need. Domain scores are generated by adding each item score and dividing by the total number of domain items.
The researcher identified and approached eligible participants after treatment completion to discuss the study and provide them with a participant information and consent form. Following informed consent, demographic and baseline (time 1) SF-SUNS questionnaires were then administered to participants. After completion of the questionnaires, participants were provided with another blank copy of the SF-SUNS accompanied by instructions to complete the questionnaire at home 5 days later and postback using the supplied reply-paid addressed envelope. Participants were advised to record the date of completion if this differed from the specified due date.
At the request of the research team's hematologist, baseline demographic and SF-SUNS data were collected from consenting participants 3 months posttreatment completion and PET scan to confirm the absence of disease. Demographic information obtained included lymphoma type, stage of disease, type of treatment received (chemotherapy +/− radiotherapy), date of diagnosis, time since diagnosis, comorbid conditions, gender, age, weight, marital status, age of children (if any), postcode, occupation, income level, education level, and health behaviors such as smoking and alcohol consumption. Participants then completed the SF-SUNS at time 2 (5 days following time 1 completion) at home.
All analyses were performed using IBM SPSS Statistics Version 25 data analysis software (IBM Corp. Released 2017. IBM SPSS Statistics for Macintosh, Version 25.0. Armonk, NY: IBM Corp). Descriptive statistics were used to analyze all data. Descriptive analyses were used to analyze and describe demographic data. To assess for absolute consistency of SF-SUNS items for test–retest reliability data, an ICC with a random-effects model was used to compare each item at time 1 and time 2. The ICC measure was chosen for its ability to discriminate between sets of scores ranked in the same order but not necessarily in agreement and adjusts for the degree of test–retest agreement expected by chance., The closer the value of the ICC to 1.0, the greater the reliability of the item or measure. The guidelines developed by Cicchetti and Sparrow  were used to determine the level of clinical significance of the ICC values obtained: <0.40 = poor, 0.40–0.59 = fair, 0.60–0.74 = good, and >0.75 = excellent. For this study, items classified as achieving “fair to excellent” reliability, ICC >0.40, were reported. Cronbach's alpha, a measure of internal consistency, was used to measure the scale reliability.
To examine the distribution of unmet needs, the five levels of unmet need were collapsed to three levels. A score of 0 (no unmet need) remained the same. Scores of 1 or 2 (low and moderate unmet need) were reclassified as 1 (low–moderate unmet need), and scores of 3 or 4 (high and very high unmet need) were classified as 2 (high–very high unmet need).
| Results|| |
There were slightly more male (n = 22, 55%) participants, and a greater number of participants with NHL (n = 29, 72.5%) compared with HL (n = 11, 27.5%) [Table 1]. This was in keeping with the current disease statistics which reflect a greater number of NHL than HL diagnoses. Almost one-third of participants were aged between 18 and 39 years (32.5%), and a greater proportion had a university qualification (n = 16, 40%) [Table 1]. Although the majority of participants were currently working (n = 15, 37.5%) and had been throughout their treatment, 30% (n = 12) were looking for work or had no return to work date set. Over half the participants had a partner (n = 25, 62.5%). Forty participants completed both time 1 and time 2 SF-SUNS. The majority of participants (n = 35, 87.5%) completed time 2 SF-SUNS 5 days after time 1 (range 4–7 days).
|Table 1: Baseline participant demographic and disease characteristics (n=40) |
Click here to view
ICCs, 95% confidence intervals, and clinical significance are shown in [Table 2]. One (3%) item met the “excellent” criteria for clinical significance; Finding car parking I can afford at the hospital or clinic. Twelve (40%) items met the “good” criteria (0.60–0.74) and 11 (37%) items met the “fair” criteria (0.40–0.59). In summary, test–retest data showed “fair” to “good” reliability for the majority of items (23/30, 77%).
Overall Cronbach's alphas were 0.92 at time 1 and 0.94 at time 2, with subscales [Table 2] ranging from 0.74 and 0.69 for information needs, 0.65 and 0.83 for work and financial needs, 0.89 and 0.85 for access and continuity of care, and 0.90 and 0.94 for coping, sharing, and emotional needs, respectively. These results support strong internal consistency for the overall scale. Item-to-total correlations between 0.40 and 0.70 indicate that items are not redundant or measuring needs similar to other items within the instrument. Using this criterion, the SF-SUNS demonstrated item-to-total correlations between 0.40 and 0.70 at time 1 for 24 items (80%) and at time 2 for 19 items (63%) [Table 2]. The majority of items were considered relevant and to be measuring unique needs.
| Discussion|| |
Our study is the first to report test–retest data for the SF-SUNS. The majority of items met absolute consistency for reliability ICC scores of >0.40 for test–retest, categorized as “fair” to “good.” An “excellent” clinical significance score was achieved for only one item (3%), related to car parking costs which are unlikely to change over time. Needs-based instruments such as the SF-SUNS measure the degree of an individual's perceived unmet need at one point in time. Importantly, Cronbach's alpha scores at time 1 and time 2 demonstrated a high degree of internal consistency and high item-to-total correlations, confirming that items in the tool were reliable.
A criterion for psychometrically sound needs-based tools is the requirement for an instrument to be responsive to changes over time.,, Although our ICC results may reflect the responsiveness of the SF-SUNS to changes in need over the data collection period, further research is required to detect clinically meaningful change for patients. All participants completed the time 2 questionnaire at home, well away from the hematology clinic where the time 1 questionnaire was completed. It is possible that participants may have had additional time to more accurately reflect on the level of unmet need. Similarly, time 1 scores may have been impacted by participants' anxiety at the hospital appointment where patients often worry about test results and potential relapse. In addition, fatigue is a recognized effect of lymphoma treatment  and may have potentially affected participant responses at either time point. Finally, most items were similarly balanced for both time points from “no unmet need” to “low unmet need” or “low unmet need” to “no unmet need.”
It is important to allow cancer survivors the opportunity to self-identify unmet needs and issues of concern. Survivorship needs-based instruments provide a consistent method for this purpose. Furthermore, it is important that any tool is responsive to change as individuals' issues, concerns, thoughts, and feelings can change from day-to-day,, particularly during survivorship transition as individuals move on with their lives after cancer treatment. Such reliable and valid instruments can facilitate individualized survivorship care and tailored support and resources.
It is important to note that the original SUNS demonstrated low test–retest reliability acceptability, with the authors suggesting that the test–retest timeframe was too long at 28 days. Since our study was part of a larger study involving an intervention group, a 5-day later test–retest assessment was deemed an appropriate timeframe to ensure completion of the time 2 SF-SUNS before the implementation of any needs-based interventions associated with the larger study. Importantly, this time period was also in keeping with the recommended 2–14-day time period for test–retest procedures.,,
A limitation of this study may have been the sample size of 40 participants, despite sample size calculations indicating that this number would be sufficient to adequately perform test–retest reliability with confidence. Many participants (n = 16, 40%) attended the baseline appointment, where time 1 SF-SUNS was administered, accompanied by a support person (partner or family member). We acknowledge that this may have influenced time 1 responses. Likewise, time 2 responses may have similarly been influenced as the SF-SUNS was completed at home. We can confirm that participants did not receive any needs-based interventions between time 1 and time 2 completion of the SF-SUNS.
| Conclusion|| |
We suggest that needs-based assessments should be used routinely during the survivorship period to facilitate survivorship care that is tailored and responsive to individuals' changing needs. Valid and reliable survivor-specific measures are essential for routine screening and follow-up. Nurses in particular are a valuable resource in the survivorship phase to assess for areas of concern or unmet needs and for the provision of information, support, and resources that are tailored to the individuals' unique needs. Further testing of the SF-SUNS is recommended in hematology and other cancer populations to further understand and demonstrate the responsiveness of this instrument to changes in need over the survivorship period.
Financial support and sponsorship
This research is supported through the University of Notre Dame Australia's Collaborative Research Network Grant and an Australian Government Research Training Program Scholarship.
Conflicts of interest
There are no conflicts of interest.
| References|| |
Howlader N, Noone AM, Krapcho M, Miller D, Bishop K, Kosary C, et al
. SEER (Surveillance, Epidemiology and End Results) Cancer Statistics Review, 1975-2014. Bethesda, MD: National Cancer Institute; 2016.
Carey M, Anderson A, Sanson-Fisher R, Lynagh M, Paul C, Tzelepis F, et al.
How well are we meeting haematological cancer survivors' preferences for involvement in treatment decision making? Patient Educ Couns 2012;88:87-92.
Sant M, Minicozzi P, Mounier M, Anderson LA, Brenner H, Holleczek B, et al.
Survival for haematological malignancies in Europe between 1997 and 2008 by region and age: Results of EUROCARE-5, a population-based study. Lancet Oncol 2014;15:931-42.
Campbell HS, Hall AE, Sanson-Fisher RW, Barker D, Turner D, Taylor-Brown J, et al.
Development and validation of the Short-Form Survivor Unmet Needs Survey (SF-SUNS). Support Care Cancer 2014;22:1071-9.
Taylor K, Chan RJ, Monterosso L. Models of survivorship care provision in adult patients with haematological cancer: An integrative literature review. Support Care Cancer 2015;23:1447-58.
Arden-Close E, Absolom K, Greenfield DM, Hancock BW, Coleman RE, Eiser C, et al.
Gender differences in self-reported late effects, quality of life and satisfaction with clinic in survivors of lymphoma. Psychooncology 2011;20:1202-10.
Boyes AW, Girgis A, D'Este C, Zucca AC. Prevalence and correlates of cancer survivors' supportive care needs 6 months after diagnosis: A population-based cross-sectional study. BMC Cancer 2012;12:150.
Hall A, Campbell HS, Sanson-Fisher R, Lynagh M, D'Este C, Burkhalter R, et al.
Unmet needs of Australian and Canadian haematological cancer survivors: A cross-sectional international comparative study. Psychooncology 2013;22:2032-8.
Hjermstad MJ, Evensen SA, Kvaløy SO, Loge JH, Fayers PM, Kaasa S, et al.
The CARES-SF used for prospective assessment of health-related quality of life after stem cell transplantation. Psychooncology 2003;12:803-13.
Lobb EA, Joske D, Butow P, Kristjanson LJ, Cannell P, Cull G, et al.
When the safety net of treatment has been removed: Patients' unmet needs at the completion of treatment for haematological malignancies. Patient Educ Couns 2009;77:103-8.
Aziz NM. Cancer survivorship research: State of knowledge, challenges and opportunities. Acta Oncol 2007;46:417-32.
McDowell ME, Occhipinti S, Ferguson M, Dunn J, Chambers SK. Predictors of change in unmet supportive care needs in cancer. Psychooncology 2010;19:508-16.
Monterosso L, Taylor K, Platt V, Lobb E, Krishnasamy M, Musiello T, et al.
Aqualitative study of the post-treatment experiences and support needs of survivors of lymphoma. Eur J Oncol Nurs 2017;28:62-8.
Taylor K, Monterosso L. Systematic review of the tools used to assess the informational and practical needs of acute leukaemia and lymphoma survivors. Aust J Cancer Nurs 2016;17:6-12.
Jiao M, Hall A, Nolte L, Piper A, Lisy K, Jefford M. A rapid review of needs assessment tools for post-treatment cancer survivors. Eur J Cancer Care 2017;e12764:1-10.
Campbell HS, Sanson-Fisher R, Turner D, Hayward L, Wang XS, Taylor-Brown J, et al.
Psychometric properties of cancer survivors' unmet needs survey. Support Care Cancer 2010;19:221-30.
Hall A, D'Este C, Tzelepis F, Sanson-Fisher R, Lynagh M. The survivor unmet needs survey (SUNS) for haematological cancer survivors: A cross-sectional study assessing the relevance and psychometric properties. BMC Health Serv Res 2014;14:211.
Taylor K, Joske D, Bulsara M, Bulsara C, Monterosso L. Protocol for care after lymphoma (CALy) trial: A phase II pilot randomised controlled trial of a lymphoma nurse-led model of survivorship care. BMJ Open 2016;6:e010817.
Swash B, Hulbert-Williams N, Bramwell R. Unmet psychosocial needs in haematological cancer: A systematic review. Support Care Cancer 2014;22:1131-41.
Oberoi D, White V, Seymour J, Prince HM, Harrison S, Jefford M, et al.
The course of anxiety, depression and unmet needs in survivors of diffuse large B cell lymphoma and multiple myeloma in the early survivorship period. J Cancer Surviv 2017;11:329-38.
Terwee CB, Bot SD, de Boer MR, van der Windt DA, Knol DL, Dekker J, et al.
Quality criteria were proposed for measurement properties of health status questionnaires. J Clin Epidemiol 2007;60:34-42.
Walter SD, Eliasziw M, Donner A. Sample size and optimal designs for reliability studies. Stat Med 1998;17:101-10.
Filsinger S, Burkhalter R, Campbell HS. Survior Unmet Needs Survey: User Guide. Ontario, Canada: University of Waterloo; 2011. p. 1433-7339.
Cicchetti DV. Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychol Assess 1994;6:284-90.
Bujang MA, Baharum N. A simplified guide to determination of sample size requirements for estimating the value of intraclass correlation coefficient: A review. Arch Orofac Sci 2017;12:1-11.
Weir JP. Quantifying test-retest reliability using the intraclass correlation coefficient and the SEM. J Strength Cond Res 2005;19:231-40.
Cicchetti DV, Sparrow SA. Developing criteria for establishing interrater reliability of specific items: Applications to assessment of adaptive behavior. Am J Ment Defic 1981;86:127-37.
Rosner B. Fundamentals of Biostatistics. 8th
ed. United States: Cengage Learning; 2016.
Leong FT, Austin JT. The Psychology Research Handbook: A Guide for Graduate Students and Research Assistants. 2nd
ed. Thousand Oaks, CA: Sage; 2006.
DeVellis RF. Scale Development: Theory and Applications. 3rd
ed. Thousand Oaks, California: Sage; 2012.
McDowell I. Measuring Health: A Guide to Rating Scales and Questionnaires. 3rd
ed. Oxford: Oxford University Press; 2006.
Streiner DL, Norman GR. Health Measurement Scales: A Practical Guide to Their Development and Use. 2nd
ed. Oxford: Oxford University Press; 2003.
Thewes B, Butow P, Zachariae R, Christensen S, Simard S, Gotay C, et al.
Fear of cancer recurrence: A systematic literature review of self-report measures. Psychooncology 2012;21:571-87.
Hawkins NA, Pollack LA, Leadbetter S, Steele WR, Carroll J, Dolan JG, et al.
Informational needs of patients and perceived adequacy of information available before and after treatment of cancer. J Psychosoc Oncol 2008;26:1-6.
| Authors|| |
[Table 1], [Table 2]