Dr. George McDonald and his colleague Dr. Reginald Lee from the University of South Florida (USF) met with the Hernando County School Board on Aug. 14, 2018 to share the results of the 2018 Superintendent’s Performance Survey. The survey, prepared by USF’s Center for Research, Evaluation, Assessment and Measurement (CREAM), received some assistance from board members Susan Duval and Linda Prescott to better address Hernando County’s specific situation and work culture. CREAM’s evaluation centered on Dr. Lori Romano, who was fired by the school board in June. Dr. Romano has requested a post-termination hearing.
This was the second year the school district employees have participated in the electronic survey, Dr. McDonald said. An email with a survey link was sent to 2,947 employees which included approximately 500 employees who were not able to participate the previous year. Of those 2,947 employees who were contacted, 938 responded. McDonald stated that responses represented various departments, from the school board members, both instructional and non-instruction school staff, and administrators at different levels.
Surveys gather demographic information from the respondents as well. In this case, how long the individual had lived in Hernando County (approximately 75%: an average of 11+ years), how long they had worked in Hernando County (approximately 50%: more than 11 years) how long they had worked for the school district (10% responded: less than 3 years). There were 52 questions with a rating system of 1-5. There were also 18 open-ended questions addressing Romano’s effectiveness as a Superintendent and areas where employees thought she could improve.
Anonymity in surveys is important. The respondent needs to be secure that their answers will not be traced back to them, even if there is a public records request. That’s something McDonald could not guarantee this year after an attorney requested and received from USF the redacted 2016-2017 documentation. McDonald stated that this year, no names or IP addresses were captured so that data provided in the report’s unredacted copy to the school board represents uncorrected responses to the open-ended questions.
McDonald said the 2017-2018 survey was determined to be highly reliable. In other words, “Does it measure what it intends to measure?” The number of respondents (938) also indicates that the data is reliable. (Any number greater than 600 shows increased reliability.) The results were stable regardless of whether the respondent had answered last year’s survey or if this was the first survey completed.
The team used the Linear Logistic Test Model, where negative numbers showed an endorsement of (or agreement with) an item. Where the respondent worked, and at what level, had an impact on the results. For example, a school administrator would look at Romano’s performance differently than an instructional staff member or a school board member.
As part of the sample provided to the school board, Dr. Lee randomized the list and then chose the top three results. On the five-point scale, an average of 4.5 = 90%, and the corresponding “logit” is seen as a negative number. Only differences greater than .5 (10-12%) were noted as significant or valid.
McDonald stated several times that the results seem to be backward from what you might expect to see, where negative numbers are good and positive numbers are not. Anything over 80% is a “good” number, he said.
If the person who is the subject of the survey is performing adequately, the scores should be around 4.0, with an average of around 80%, meaning the person being evaluated was “effective” or “very effective.” An exceptional performance should show at least 4.5 or greater.
CREAM’s results show that there “was not a statistically different change” between the 2017 and 2018 surveys. None of the results are below -.073. McDonald stated there is no significant change in Romano’s performance from one year to the next.
The survey covered nine areas, or domains. Communication, Community relations, and Curriculum Planning and Development, showed the greatest weakness (poorest performance) for Romano. Values and Ethics of Leadership had the highest performance, and Leadership and District Culture also had higher scores. Labor relations had almost no change.
Overall, the highest rated responses did not reach the 4.0 mark but hovered between the 3.0 – 3.5 range. The lowest rated responses were generally between 2.0 And 2.5. As noted above, the responses related to Romano’s communication with the community, media, and others.
Across the nine domains, McDonald stated the respondents were given a choice of marking Romano as “effective” or “needs improvement,” and some chose to add comments as well. While the chart appeared to show that the majority of respondents indicated that Romano needed to improve in most areas, McDonald’s phrasing was that Romano was “improved.”
“Being effective doesn’t necessarily mean positive,” McDonald said. “Needs improvement always needs improvement.” The comments of those who responded were provided to the school board so they could read and understand why selecting “effective” could be seen as negative.
Leadership and District Culture: Romano received high marks from district staff and administrators, but the marks from both instructional and non-instructional staff were very low by comparison. “Where you work and what you do,” McDonald said, “has a bearing on what you believe. It matters what role you have.”
Policy and Governance: Again, district administrators and staff rated Romano highly, but those who worked with students directly or were non-instructional staff, felt she needed a lot of improvement in this area.
Communications and Community Relations: Across the board, respondents saw that this was an area where Romano needed improvement.
Organizational Management: Both district staff and instructional staff felt that Romano was lacking in this area. School and district administrators as well as non-instructional staff found this a positive area for Romano.
Curriculum Planning and Development: All levels of staff, from non-instructional to district administrators, felt Romano needed great improvement in this domain. Though McDonald stated that the scores did not meet the criteria of a 0.5 statistically significant change, it does represent that she performed poorly here when compared to last year.
Instructional Leadership: Romano received the most positive responses here from school administrators and from some instructional and district staff.
Human Resources Management: Romano scored poorly in this domain as well. School administrators, even more than the district staff and instructional staff, felt this was an area where she needed improvement.
Values and Ethics of Leadership: Romano was perceived as being highly effective in this area. School administrators gave Romano the highest marks of all those who responded here, but she received good marks from all areas of staff.
Labor Relations: The only positive scores here came from district staff and district administrators. The response from school administrators reflects a change from last year, as there were no positive responses at all.
Over the course of the two evaluations, Romano did not show any real or statistical improvement, according to McDonald. For 2017, Romano’s overall rating was 2.86 and in 2018 her rating was 2.1, or approximately 56% endorsement (approval). The issues at Moton Elementary seemed to be the subject of many comments.
Using the nine domains as a measurement of the effectiveness of a superintendent could be beneficial to the school board. “People in Hernando are extremely honest,” McDonald said. “They tell you what they think. That gives the Superintendent the opportunity to sit down and say, ‘Here’s my strengths and here’s my areas to improve.’”
To use the same domains as a tool to evaluate the current Superintendent would give a more realistic and individualized review of whether the Superintendent was meeting the goals or not. Dr. Lee stated that the Superintendent would be judged on his own merits and not as a comparison to Dr. Romano.
“You don’t want people to be fearful of an evaluation,” Lee said. “You want them to use the information. It gives the people in your district an opportunity to tell you what they think. It’s up to you to decide how to use that information.”
Though the survey is a good tool, Lee cautioned that it should not be the only tool that the school board uses to evaluate or measure the Superintendent’s performance. The board members did express interest in working with Drs. Lee and McDonald for the upcoming year, subject to approval.
