Description

Reports to: Director of Certification Development, CAS

Scope of Responsibilities

The Certification Analyst is responsible for supporting an examination scoring process to ensure that it is reliable, fair, and valid and meets testing industry standards. The position will work closely with the CAS Certification team, exam working groups and task forces, and 3rd party testing partners. The Certification Analyst conducts activities across the CAS certification development lifecycle, including but not limited to item and test analysis, assessment scoring, equating, standard setting, and other data activities. The Certification Analyst uses data and business solutions to enhance best practices within the admissions department and across the CAS.

 Duties and responsibilities include but are not limited to the following:

Assessment Development 

•        Support practice analyses projects. 
•        Conduct standard setting projects.
•        Manage grader volunteer teams and provide guidance on item rubric best practices within the item banking software.
•        Manage and facilitate the item/content grading training, grading summits/meetings, and any additional exam development workshops. 
•        Support Certification Program Manager in analyzing the performance of items to inform item development workflow.

Assembly and Production

•        Set up assessment grading rubrics and solution files within the grading software.
•        Manage user acceptance testing guidelines for all scoring related activities.
•        Manage the exam/assessment grading process with the grading software vendor and other CAS departments or 3rd party vendors. 
•        Maintain the security of item bank, exam forms, and confidential candidate/vendor information.

Scoring, Analysis, and Reporting

•        Run programs and interpret data to process assessment results.
•        Maintain equating mechanism across the program.
•        Maintain quality control checks on overall scoring and analysis processes.
•        Prepare reports summarizing assessment results.

       Develop and maintain dashboards including but not limited to:

o   Exam registrations.
o    Exam delivery statuses.
o   Item statistics and analysis/psychometric support.
o   Exam metrics on delivered exams.
o   Grading metrics on items and exam performance.

•        Work with vendors to ensure exam security and manage exam security breaches.
•        Data Responsibilities

o   Maintain data integrity.
o   Design queries, compile data, and generate daily/weekly reports.
o   Create data charts and graphs for reporting purposes.
o   Perform data collection and entry as needed.
o   Document data processes.
o   Interpret multiple sets of data with different visualizations for different stakeholders.
o Develop and maintain a robust security program with support from the Exam Developer. 

Operational Support

•  Maintain all sections related to grading/scoring in the Exam Development Processes and Policies document.
• Maintain written instructions and standard operating procedures for accomplishing key certification functions.
• Monitor and analyze program specific performance metrics.
• Handle other projects and assist other operating areas of the CAS Office as assigned.

Requirements

Required Experience and Qualifications

•  Masters’ degree in evaluation and testing, psychometrics, statistics, education, educational psychology, or similar field
• Three years of statistical/data analysis or equivalent required.
• Experience with examination development highly preferred.
• Experience with ISO 17024 standards for certification programs preferred.
• Ability to plan, track and perform multiple tasks, manage multiple priorities and maintain consistent attention to detail in the IT space.
• Excellent communication skills to respond to and interact with the certification team, volunteer working groups, task forces, and candidates.
• Excellent time management skills required
• Experience in project, vendor, and volunteer management.
• Advanced proficiency using Microsoft Office applications.
•  Knowledge of item banking software. Proficiency in ExamDeveloper software is a plus.
• Knowledge of computer applications for database management and data analysis. Proficiency in Cobalt is a plus.
• Ability to travel – up to 10%.

• Measures of accountability:
  Delivers in accordance with CAS project timelines and annual performance goals.