Chapter 1. Introduction

This document presents the test scenarios that will be used to verify reference implementation of Open Source Software for Intelligent Transportation Systems (OSS4ITS) software suite to demonstrate the full functionality of the software system as part of the OSS4ITS project sponsored by the Office of Operations Research and Development (HRDO), Federal Highway Administration (FHWA). The Leidos team will confirm the performance standards are met as documented in this test plan based on project documentation and input from team members.

Purpose and Objective

This Reference Implementation Test Plan presents the test methods that shall use OSS4ITS Software suite of applications including V2X Hub, ODE, SDC and CARMA components e.g., CARMA Platform, CARMA Cloud, and CARMA Messenger to demonstrate the full functionality for the Work Zone Data Exchange use case. Leidos will establish connections between the Work Zone Data Collection tool, CARMA Cloud, CARMA Platform, V2X Hub, Operation Data Environment, and SDC as defined in the OSS4ITS architecture and will stand up the reference implementation on the TFHRC Cloud. This plan ensures hardware, software and functional requirements have been met in order to demonstrate the Work Zone Data Exchange use case successfully.

This plan may be modified to incorporate comments or contributions or findings from FHWA and the OSS4ITS software team. It is expected that the use case may evolve through the course of design, development, implementation, and testing. Therefore, the Test Plan should be modified to capture the changing requirements during the agile development life cycle. In addition, the test cases defined here may also be later refined as needed based on the outcome of the verification testing.

TEST VERIFICATION AND SCOPE

The scope of this document is limited to testing of the vehicle features and infrastructure applications under the Work Zone Data Exchange use case. This test plan represents the latest agreed-to scenarios with FHWA and Leidos. In addition, this test plan documents the test approach, methodology, roles and responsibilities, schedule, equipment, facilities, and test cases for each work zone scenario.

Leidos will use the fleet of FHWA automated passenger cars to verify functionality for the Work Zone Data Exchange use case.

The Leidos team shall work together with FHWA to gather their inputs into this plan. Leidos shall respond to answer any questions and incorporate any changes resulting from peer reviews or discussions.

The success of the verification testing will be based on the successful demonstration of the individual use case scenarios without any severity one issues. All issues must be addressed, and the resolutions agreed upon. Table 1 below shows the task breakdown and participation of various teams in executing these tasks.

Table 1. Organization Task Breakdown

Task	Leidos	FHWA
Verification Test Plan Development	X
Verification Test Plan Peer Review	X	X
Verification Testing Logistics and Coordination	X
Verification Testing Readiness Review	X	X
Verification Testing Execution	X
Verification Test Results Documentation	X

Chapter 2. Test Approach

Leidos is working with FHWA to identify the list of features that need to be verified and tested for each of the Work Zone Data Exchange use case scenarios. Each test case shall include a high-level description of the use case scenario, its ODD, its targeted test facility, its sequence of events, and it’s high-level list of features to be tested. It will describe the cooperation that can take place between an equipped vehicle and other vehicles, roadside equipment, and Transportation Management Services (TMS).

In addition, Leidos reviewed the National Highway Traffic Safety Agency (NHTSA) pre-published version for “A Framework for Automated Driving System Testable Cases and Scenarios” for ADS test case/evaluation guidance and approaches. From this NHTSA documentation, the test case template has been updated to include the following elements, specific to each test scenario based on the test site availability.

Operational Design Domain (ODD) – An ODD describes the specific operating domain(s) in which an ADS feature is designed to function with respect to roadway types, speed range, lighting conditions (day and/or night), weather conditions, and other operational constraints. ODD will likely vary for each ADS feature, even if there is more than one ADS feature on a vehicle.

Object and Event Detection and Response (OEDR) – OEDR refers to “the subtasks of the DDT that include monitoring the driving environment (detecting, recognizing, and classifying objects and events and preparing to respond as needed) and executing an appropriate response to such objects and events (i.e., as needed to complete the DDT and/or DDT fallback)” (SAE International, 2016).

Leidos shall provide the final draft of the verification test plan to GTM for review, in order to communicate the final features being developed for each Work Zone use case and potentially leveraged to develop their own validation test plan.

Documentation

A Verification Test Report shall be compiled to summarize the test results and identify outstanding issues. Failures, as well as any deviations or workarounds, will be recorded.

Assumptions and Constraints

The bulleted list below explains the assumptions and/or constraints expected prior to the testing or demonstration:

Test personnel have at least a basic understanding of the CARMA platform in both configuration and troubleshooting
Test personnel are properly trained and licensed to operate CARMA vehicles
The test is conducted in at a legal, controlled, and safe environment

Test Evaluation

Each test case shall be recorded with a pass or fail in the test report. All failures, as well as any deviations from the procedure or workarounds, will be recorded as well with severity levels defined below. Each test case needs to pass based on back to back testing or 2 out of 3 runs, unless otherwise specified.

Severity	Definitions
1	Prevent the accomplishment of an essential/critical capability. Causes severe disruption, and results in the permanent absence of one or more required features of the system, and no work around is known. Jeopardize safety, security, or another requirement.
2	Adversely affect the accomplishment of an essential capability but a work-around solution is known. Causes major disruption of access to one or more required features of the system.
3	Causes minor disruption of access to one or more required features of the system.
4	Result in user/operator inconvenience or annoyance but does not affect a required operational or mission-essential capability.

As mentioned in the scope, this test plan represents the latest agreed-to-scenarios with FHWA and Leidos. Therefore, there may be some deviations between the test plan and requirements for the individual use cases and scenarios.

Chapter 3. Test Schedule and Personnel

This section contains a high-level test schedule, personnel required to execute the tests, and a description of several documents that should be used to record test activities and results.

Test Schedule

The verification testing of each of the Work Zone Data Exchange use case scenarios shall occur at USDOT TFHRC as agreed upon with the client. The targeted dates and order of the verification testing of each of the use cases could be adjusted based on development schedule and particular circumstances.

Table 2. Work Zone Data Exchange Scenario Testing Schedule

Scenario	Test Facility	Leidos Verification Testing	Validation Testing
Work Zone Management (WZM)	TFHRC	TFHRC	TFHRC

The daily schedule of activities during verification testing shall be discussed at least a week prior to the event. The schedule shall describe the anticipated daily activities, including an estimated duration and assignments of each task, if possible. During the week of testing, there shall be a daily team wrap up to discuss the accomplishments, issues and plans for the day. The schedule will be reviewed or adjusted daily based on the team’s progress. Some activities may not be necessary or may be carried out concurrently. The test schedule assumes no interruption of testing activities due to weather or any other unexpected delays.

An example schedule can be seen below. The test team should move down the test plan from top to bottom. This will ensure that new functionality is tested only after the technologies it depends on are deemed safe and acceptable. The below schedule only serves as an initial sample guide and does not account for changes in testing conditions such as needing to switch locations or adverse weather. A separate spreadsheet shall be used for the official schedule.

Table 3. Example Daily Testing Schedule

Time	Monday	Tuesday	Wednesday	Thursday	Friday
8am – 9am	Arrival and setup at the test facility. Review testing prioritization and schedule. Safety Inspection	Review testing prioritization and schedule. Safety Inspection	Review testing prioritization and schedule. Safety Inspection	Review testing prioritization and schedule. Safety Inspection	Review testing prioritization and schedule. Safety Inspection
9pm - 4pm	Verification Testing	Verification Testing	Verification Testing	Verification Testing	Verification Testing
4pm – 5pm	Daily Tag-up to review results and action items	Daily Tag-up to review results and action items	Daily Tag-up to review results and action items	Daily Tag-up to review results and action items	Daily Tag-up to review results and action items Departure and tear down at the test facility.

The procedures for the safety inspections shall be developed prior to the test event on a separate document. The safety inspection shall include the in-stock and after-market hardware visual inspections, sensor calibration checks, vehicle controller and application safety overrides, and test facility operational basic safety procedures.

Personnel

The following staff is anticipated to complete the activities in the estimated amount of time listed in Chapter 3. Test Schedule and Personnel. Listed below are the general personnel required for executing the test verification plan and documenting its results.

NOTE: Some roles except for the Vehicle Operator/Safety Driver may be combined such that a single person can assume multiple roles for an individual test case.

Test Lead

The designated Test Lead will make final decisions on pass/fail criteria of individual test cases, based on reports from the Data Recorder/Tester. Additionally, the Test Lead will be responsible for overseeing the overall test schedule, prioritizing the tasks and test cases, and communicating support needs with supporting organizations.

Vehicle Operator/Safety Driver

The Vehicle Operator will be responsible for safe operation of the vehicle throughout the test runs, including during test set-up, experimental runs, and post-test activities. Vehicle Operators shall hold a valid driver’s license for the class of the vehicle, may be trained in advanced vehicle control techniques, and shall be oriented to the special situations they will face during testing. The Vehicle Operator shall expect to receive instructions on test execution and expectations from the Tester and shall report any confusion or concerns both before and during the tests to all involved parties.

The operator will wear a seat belt at all times, remain alert for direction from Traffic Safety Controllers on the roadside (if any), and be responsible for compliance with traffic law at all times. The operator shall refrain from using cell phones or other handheld wireless communication devices while driving, except for communications with others in the test team directly related to the execution of those tests. The operator will be responsible for confirming the selected parameters to the Data Recorder for each test run, validating the entrance criteria, verifying the readiness of test equipment, and announcing the start and end of each test run. The operator will narrate any relevant observations during testing for the Data Recorder to ensure that any test irregularities or results are properly documented.

At all times, the operator will be responsible for navigating incidents, exceptions, cancellations, or re-scheduling tests in the event of a failure that prevents a test from being executed.

Passenger/Observer

The Passenger may occupy a seat of the test vehicle during testing activities to monitor the user interface of the application. The Passenger will wear a seatbelt at all times and remain alert for any discrepancies in the expected application performance and notify the operator if a manual override is necessary.

Data Recorder

The Data Recorder is responsible for recording the outputs and overall results of each test. The data recorder will wear a seatbelt at all times and ride in either the back or front of the vehicle with a copy of the test cases. The data recorder will read out test instructions and write down results as dictated by the operator. The Data Recorder shall be responsible for making an initial assessment of whether the test was successful. However, the data recorder will provide all observations to the Test Lead at the end of the test day for final confirmation of test success or resolution of failed tests.

Tester

The Tester will coordinate the different scenarios or operations that need to be tested during each run to each Vehicle Operator. The Tester shall confirm all the configuration parameters that need to be implemented on the vehicles before the start of each run. Additionally, the Tester shall be responsible for instructing the Vehicle Operator on test execution plans and expectations of results. The Tester can choose to sit at the back or front of the vehicle and can also assume the role of Passenger and/or Data Recorder.

Software Engineer

The Software Engineer is responsible for supporting any software or configuration issues during the test run. The Software Engineer can be asked to ride in the vehicles to support testing and troubleshoot any issues/anomalies on the spot. The Software Engineer can choose to sit at the back or front of the vehicle, and can also assume the roles of Passenger, Tester and/or Data Recorder. During debugging/troubleshooting, the Software Engineer can redeploy software or change configuration parameters before the start of each run.

Traffic Safety Monitor

If a test is occurring in an area where traffic may be present, then a project team member will work outside of any test vehicle, wear a reflective vest, and block or monitor traffic as necessary for the test to be executed safely. This person must have an easy means of communicating with someone inside the vehicle to help coordinate test execution such as to alert the safety driver of a conflict using a 2-way radio.

Chapter 4. Test Environment

The FHWA has selected TFHRC as the site to conduct the Work Zone Data Exchange use case scenario demonstration as described in Chapter 3 Test Schedule and Personnel.

Although use cases studied may cover a spectrum of conditions, the Leidos team will limit the testing on closed test tracks, at daylight and in mild weather. At this time, there are no plans of performing any of the described tests on public roads.

Prior to conducting tests, the test environment should be inspected for safety, all hardware and software required to execute the test cases are readily available, and no other barriers exist that would prevent a proper demonstration of the vehicle capabilities.

For the physical and safety inspections, the vehicle should be positioned in the well-lit area and stationery in order to visually inspect the interior and exterior of the vehicle’s in-stock and after-market hardware installed prior to conducting the tests.

Test Facility

The Turner-Fairbank Highway Research Center (TFHRC) has been selected as the test facility for these tests. Its ODD constraints are described in the subsections below.

Name: Turner-Fairbank Highway Research Center (TFHRC)
Address: 6300 Georgetown Pike, McLean, VA 22101

Source: Google

Figure 1. Turner Fairbank Highway Research Center Campus

ODD – Physical Infrastructure

ODD Elements	Examples
Roadway Types	two-lane, one way taper work zone
Roadway Surfaces	asphalt
Roadway Edges and Markings	shoulder (paved), curb, cones
Roadway Geometry	straightaways, curves, hills

ODD – Operational Constraints

ODD Elements	Description
Minimum Speed Limit	5 MPH
Maximum Speed Limit	25 MPH
Traffic Density	Low traffic
Automated Vehicles	Allowed

ODD – Objects

ODD Elements	Description
Signage (Signs (e.g., stop, yield, pedestrian, railroad, school zone, etc.), traffic signals (flashing, school zone, fire department zone, etc.), crosswalks, railroad crossing, stopped buses, construction signage, first responder signals, distress signals, roadway user signals, hand signals)	Cones or Construction Equipment (if available), Dummy Construction Workers (if available)
Roadway Users (Vehicle types (cars, light trucks, large trucks, buses, motorcycles, wide-load, emergency vehicles, construction equipment, horse-drawn carriages/buggies), stopped vehicles, moving vehicles (manual, autonomous), pedestrians, cyclists)	Any vehicles
Non-roadway User Obstacle/Objects (Animals (e.g., dogs, deer, etc.), shopping carts, debris (e.g., pieces of tire, trash, ladders), construction equipment, pedestrians, cyclists)	None expected or shall be used during this testing.

ODD – Environmental Conditions

ODD Elements	Description
Weather (e.g., wind, rain, snow, sleet, temperature)	Clear and calm (Preferred scenario) Light rain is tolerable Light wind is allowed, but high-speed gusts must be avoided. No falling snow or sleet or icy roads A small amount of liquid water, slush or packed snow on roadway is acceptable
Weather-induced Roadway Conditions	None
Particulate Matter (e.g., fog, smoke, smog, dust/dirt, mud)	None. Clear visibility.
Illumination	Day

ODD – Connectivity

ODD Elements	Description
Vehicles	Cellular, DSRC, C-V2X allowable
Traffic Density Information (e.g., crowdsourced data (e.g., Waze) and V2I)	N/A
Remote Fleet Management System (supported by remote operation)	N/A
Infrastructure Sensors and communications (e.g., WZ alert, VRUs, GPS, 3-D high-definition maps, weather data, data on cloud)	RSU, GPS, 3-D high-definition maps

ODD – Zones

ODD Elements	Description
School/Construction Zones	N/A
Traffic Management Zones	Work Zone
Geo-Fencing	TFHRC campus

Test Vehicles

One CARMA vehicle is ADS Level 3+ capable as enabled by the CARMA Platform and equipped with an OBU to exchange messages with other vehicles and CARMA Cloud. It has a human machine interface that indicates current status and alerts to the vehicle operator and test engineer.

More information about these vehicles can be found on the CARMA Confluence page, specifically in the CARMA Vehicle Specification Brochure found in the Brochures section (link can be found in the references section).

Chapter 5. Test Data Collection, Analysis and Processing

Refer to the CARMA V3 Data Management Plan (DMP) that describes what data will be acquired or produced, how data will be collected and managed, and what mechanisms will be used to appropriately share and preserve the data with USDOT during and after the course of the research project. This DMP identifies the organizations, tools, and functions necessary to carry out effective data management for its stakeholders.

Chapter 6. Test Scenario Overview

This chapter provides an overview of the Work Zone Data Exchange use case to be executed, in order to establish the general approach to verifying each scenario.

Overview

Title: ADS-Equipped Vehicle Navigating a Two-Lane, One-Way Traffic Taper Work Zone

The subject of this situational analysis is an ADS feature who’s specified ODD includes operation of ADS-equipped vehicles safely navigating a work zone on a two-lane roadway with one lane blocked. A work zone will be designated on a one way two-lane roadway on the WZDC tool. A segment of the travel lanes shall be blocked with cones used to signify the construction area. The CARMA vehicle shall start from a predetermined static starting location before the work zone and engage CARMA “Route Following” by selecting a route. The vehicle will travel in the direction towards the work zone and begin broadcasting its route via DSRC in the form of a MobilityPath message and a Traffic Control Request for traffic rules. This TCR will be received by the RSU, sent to V2X Hub, which will forward it to CARMA Cloud. CARMA Cloud will return all known traffic rules associated with incidents (including those from the WZDC tool) and return these to V2X Hub in the form of TCMs. V2X Hub then sends the TCM to the RSU which is broadcast via DSRC. These TCMs are received by the vehicle and the route is adjusted to avoid the blocked lane. The vehicle will continue through the work zone, navigating around the closed lane. As the vehicle travels through the work zone, it will be broadcasting BSMs. These BSMs will be broadcast via DSRC, received by the RSU, and forwarded to the V2X Hub. The V2X Hub then forwards these BSMs to ODE, which then sends them to SDC. SDC receives these BSMs and converts them to a kml file to be viewed.

ODD Characteristics

In addition to the test facility ODD, the following supersedes:

Maximum Speed limit of 25 MPH
Maximum work zone speed limit of 10MPH
Two-lane, one-way arterial street (or similar)
An HD map describing the normal road geometry and traffic controls

OEDR Characteristics

Objects
- Cones or Construction Equipment (if available)
- Dummy Construction Workers (if available)
Signs and Signals
- Traffic Signal
- Geofencing

Key Features

WZDC Tool

Generate work zone configuration in the WZDC web interface.
Download configuration to vehicle to drive work zone to generate GPS “breadcrumb” and mark lane closures and workers present.
Upload breadcrumbs to the WZDC web interface.
Verify work zone lane closures and workers present and publish.

CARMA Cloud

Work Zone Management RSM (XML format) download from WZDC tool REST endpoint.
TCMs generated from RSM XMLs and displayed on web interface

CARMA Platform

The CARMA vehicle broadcasts a TCR message from its OBU via DSRC containing the bounding box of the route it plans to drive.
The CARMA vehicle receives the TCM message in response to the TCR, reduces its speed appropriately and drives around the work zone area, switching lanes to the lane that is open. The following features from CARMA Core ADS will be displayed:
- Cooperative Lane Coordination (CLC) – Lane Change

V2X Hub

Receives TCR messages from CARMA vehicle and forwards them to CARMA Cloud
Receives TCM messages from CARMA Cloud and forward them to RSU.
Receives BSM messages from the vehicle via the RSU and forward them to ODE.

ODE (Operational Data Environment)

Receives BSM messages from the V2X Hub and sends them to SDC.

SDC (Secure Data Commons)

Receives BSM messages from ODE for analysis and KML file generation.

Failure Behaviors

Unable to demonstrate the features listed above.
Unable to demonstrate the sequence of events described below.
Severity one issues or anomalies occur that prevent the proper execution and completion of the use case demonstration.

Test Route

The test route for this use case at TFHRC can be found in Figure 2.

Source: V2X TMC Data Collection Website

Figure 2. Work Zone Management Scenario Testing Route

Equipment

One CARMA vehicle
Cones or dummy construction equipment
RSU on the infrastructure, connected to local network
V2X Hub, connected to the local network and CARMA Cloud

Test Cases

Test Case RI-01

Test Case #	RI-01
Test Case	Work Zone Data Collection Tool Configuration
Objective	Create, verify, and publish a work zone on the WZDC tool
Entrance Criteria	The WZDC tool is online and accessible There are no active work zones on the selected Test Route A vehicle equipped with a GPS is available to drive the Test Route A laptop is available with the Work Zone Data Collection Toolset installed and the GPS is connected to the laptop
Data Inputs	Work zone details, GPS data
Data Outputs	RSM XML
Exit Criteria	The work zone is published and the RSM xml is accessible
Test Procedures	Step Description	Expected Outcome
1	Navigate to the WZDC tool website: neaeraconsulting.com/v2x_home	The WZDC tool website is displayed (V2X TMC Data Collection Website)
2	Click Configuration Creator under V2X WorkZone Data Exchange	The Configuration Creator is shown
3	On the WZDC tool website under “Configuration File”, fill in “Work Zone Description” and “Road Name” and verify that the file name matches the file name updates accordingly.	Work zone description and road name are filled in and the file name is correct.
4	At the bottom of the screen, click the Next button.	The Configuration Data tab is displayed.
5	Under “Lane Information”, set the following values: Number of Lanes: 2 Vehicle Path Data Lane: 1	The Lane Information values are set.
6	Under “Speed Limits”, set the following values: Normal Speed: 30 At the Ref. Point: 20 When Workers are Present: 10	The Speed Limits values are set.
7	Under “Start Date”, select today’s date and under “Start Time” select a time before the current time.	The Start Date and Start Time are set.
8	Under “Days of Week”, select all days.	The Days of Week value is set.
9	Under “End Date”, select tomorrow’s date and under “End Time” enter 12:00:00.	The End Date and End Time are set.
10	Click the Next button.	The Map Location tab is displayed.
11	In the search box, type TFHRC and select the first option.	The map is centered on TFHRC.
12	Click on the map at the desired start location to place the beginning marker for the workzone and click again to place the end marker.	The workzone start and end points are set along a road at TFHRC.
13	Click the Next button.	The Additional Information tab is displayed.
14	Under “Event Status” select Active. The direction property is to describe the direction of traffic. Other fields on this page are optional and can be filled if needed.	The Event Status isset to active.
15	Set the Direction property to the desired direction of traffic.	The Direction is set.
16	Click the Next button.	The Lane Options tab is displayed.
17	Under “Lane Types”, set Lane Number 1 to left-lane and Lane Number 2 to right-lane.	The Lane Types are set.
18	Click the Next button.	The Metadata tab is displayed.
19	Fill in the required fields (as shown by a red asterisk).	The required fields under metadata are filled out.
20	Click the Save button at the bottom of the screen.	The work zone configuration is saved and the user is returned to the Configuration File tab.
21	Under the existing configuration files in the left panel, select the created work zone and click the Publish button at the bottom of the screen.	The work zone configuration is published and appears in the right panel.
22	Under the published configuration files in the right panel, select the published work zone and click the Download File button at the bottom of the screen.	The configuration file json is downloaded.
23	Get into the vehicle and connect the GPS to the laptop.	The GPS is connected.
24	On the laptop, open a console window and navigate to the Work_Zone_Data_Collection_Toolset/Work Zone Data Collection Tool directory.	The Work Zone Data Collection Tool directory is opened and contains the python WZDC_tool.py.
25	Run the script with python WZDC_tool.py using the command: python WZDC_tool.py	The WZDC tool is started.
26	Click “Choose Local Config File” and select the downloaded configuration.	The configuration file is uploaded and the tool indicates “GPS DEVICE IS FOUND”.
27	Drive the vehicle to a point well before the work zone begins.	The vehicle is stopped well before the work zone.
28	Click “Begin Data Collection” and drive in the open lane of the work-zone.	Bread crumb data is collected and saved when the vehicle enters the work zone
29	Click the Lane 2 button at the beginning of the desired lane closure and again and the end of the desired lane closure.	The lane closure is captured.
30	Continue driving until the end of the work zone is reached.	When the vehicle reaches the end of the work zone, the WZDC tool will exit and save the bread crumb data to a csv in the WZ_VehPathData directory.
31	On the WZDC website, navigate to the Upload tab and click the Upload button to upload the collected csv data.	The csv file is successfully uploaded to the website.
32	On the WZDC website, navigate to the “Verification” tab, select your work zon, and click load visualization.	A RSM map should pop up and show the work zone information.
33	If the work zone needs to be edited, make the appropriate changes and click Verify Markers, Confirm Edit, and Save Work Zone.	The work zone data should be successfully published.
34	Click Verify Work Zone Data and Publish.	The work zone is published
35	On the WZDC website, go to the “Published” tab and select the published work zone.	The published work zone is displayed.
36	Check the box for XML Roadside Safety Message and click Download Workzone Data.	The RSM XML is downloaded.

Test Case RI-02

Test Case #	RI-02
Test Case	Work Zone Data Collection Tool to CARMA Cloud Interface
Objective	Verify CARMA Cloud can receive RSM XMLs from the WZDC tool and display them on the CARMA Cloud interface
Entrance Criteria	The WZDC tool is online and accessible by CARMA Cloud CARMA Cloud is online and accessible by the WZDC tool There is an active work zone in the WZDC tool
Data Inputs	RSM XML
Data Outputs	CARMACloud lane closure visualization
Exit Criteria	The work zone is visible on the CARMA Cloud interface
Test Procedures	Step Description	Expected Outcome
1	Navigate to the “Published” tab of the WZDC website, find the created work zone and select it.	The created work zone is correctly visualized on the website map.
2	Start an SSH session with CARMA Cloud.	A SSH session is opened with the CARMA Cloud instance.
3	Open the log file /opt/tomcat/logs/carmacloud.log search for the published work-zone name.	RSMs for the created work zone are included in the logs when they are imported.
4	Navigate to the carma-cloud webpage (https://www.carma-cloud.com/) and login.	The CARMA Cloud interface is shown
5	Verify the created work zone is shown on the CARMA Cloud map.	Lane closure is present on the CARMA-Cloud web page.
6	Click on Edit mode and select the created work zone	A popup is display showing the blockage is of type RSM

Test Case RI-03

Test Case #	RI-03
Test Case	CARMA Cloud and CARMA Platform communication through V2X Hub
Objective	Verify CARMA Cloud can receive a TCR from CARMA Platform and respond with a TCM .
Entrance Criteria	A CARMA Cloud instance is running and is accessible by the V2X Hub via the required tunnels A CARMA vehicle is available and capable of running a route within range of the RSU A closed lane is configured within the desired route using the CARMA Cloud interface (closure must be added within the last 24 hours) An RSU is running and configured to forward received DSRC TCRs and TCMs to port 5398 of the V2X Hub The following DSRC Message Manager Plugin values are set: Destination_1: <IP of RSU>:1516 Messages_Destination_1 = …{ "TmxType": "TMSG05-P", "SendType": "TMSG05-P", "PSID": "0x8003" }… The following Message Receiver Plugin values are set: IP = IP of V2X Hub Port = 5398 The Message Receiver plugin is enabled The CARMA Cloud plugin is enabled
Data Inputs	CARMA TCR message
Data Outputs	CARMA TCM message
Exit Criteria	CARMA TCM messages are received by the CARMA Vehicle
Test Procedures	Step Description	Expected Outcome
1	Startup the CARMA platform on the vehicle and drive it to a place on the desired route within range of the RSU but do not select a route	The CARMA vehicle is within range of the RSU but no TCRs are broadcast because a route has not been selected
2	Select a route on the CARMA platform	A route is successfully loaded and the vehicle starts to broadcast TCRs via DSRC
3	Open the Messages tab on V2X Hub and verify an entry for the sent TCR is added and the Count increases (show as TMSG04-P)	An entry for TMSG04-P exists and increments in Count
4	On the CARMA vehicle, enter the following command: rostopic echo /messages/outgoing_geofence_control	Outgoing TCRs are show on the output as they are sent by the vehicle
5	In Messages tab on V2X Hub and verify an entry for the TCM is added and the Count increases (shown as TMSG05-P)	An entry for TMSG05-P exists and increments in Count
6	On the CARMA vehicle, enter the following command: rostopic echo /messages/incoming_geofence_control	TCMs are show on the output as they are received by the vehicle

Test Case RI-04

Test Case #	RI-04
Test Case	CARMA Platform Work Zone Navigation
Objective	Verify a the CARMA vehicle can navigate around the work zone using the TCR from CARMA Cloud and V2X Hub can receive BSMs and forward them to ODE as Kafka messages
Entrance Criteria	An OBU is configured and broadcasting BSMs within range of the RSU An ODE instance is running and is accessible by the V2X Hub and the ODE web interface is accessible (<ip of ODE>:8080) The following Message Receiver Plugin values are set: IP = IP of V2X Hub Port = 26789 An RSU is running and configured to forward received DSRC BSMs to port 26789 of the V2X Hub The Message Receiver plugin is enabled and receiving BSMs The following ODE Logger Plugin values are set: KafkaBrokerIp = the IP of the ODE instance
Data Inputs	TCM, BSM
Data Outputs	Kafka BSM messages
Exit Criteria	The CARMA vehicle drives the Test Route and navigates around the work zone. V2X Hub receives the vehicle’s BSMs and sends them to ODE.
Test Procedures	Step Description	Expected Outcome
1	Verify the CARMA vehicle is broadcasting BSMs	An entry for BSMs is visible in the Messages tab of V2X Hub and the count is increasing
2	Open the ODE Demonstration Web console	The ODE Demonstration Web Console is opened
3	Click the Disconnect Websocket button	The websocket is disconnected and the Disconnect Websocket button is grayed out
4	Click the Connect Websocket button	The websocket is connected and the Connect Websocket button is grayed out
5	Under Listen for Kafka Messages, click the Clear Messages button	The Kafka Message box is cleared
6	Open an SSH session to the ODE instance and enter the command “docker ps”	The following containers are shown as running: jpo-ode_ode_1 jpo-ode_kafka_1 jpo-ode_zookeeper_1 jpo-ode_cvpep_bsm_depositor_1
7	Enable the ODE Logger plugin and record the current time	The ODE Logger plugin is enabled and the indicator light turns green. The time is recorded
8	Observe the BSM Kafka messages appear in the ODE Demonstration Console	BSM Kafka messages appear in the ODE
9	Engage the CARMA vehicle into autonomous driving mode	The vehicle begins to drive the Test Route
10	Verify the CARMA vehicle changes lanes to avoid the work zone	The CARMA vehicle successfully changes lanes before the work zone
11	Verify the CARMA vehicle returns to the starting lane once it has passed the work zone	The CARMA vehicle returns the starting lane after it has passed the work zone
12	Disable the ODE Logger Plugin and record the current time	The ODE Logger plugin is disabled and BSMs stop being sent to ODE. The time is recorded

Test Case RI-05

Test Case #	RI-05
Test Case	Secure Data Commons
Objective	Verify the Secure Data Commons can receive data from ODE and convert it to a kml file
Entrance Criteria	BSM data has been sent to ODE An ODE instance is running and is accessible by the V2X Hub and the ODE web interface is accessible (<ip of ODE>:8080) The tester has access to SDC and BSM_to_KML.py is on their network Python is installed on the SDC workstation.
Data Inputs	BSM JSON format files
Data Outputs	KML format files and visualization in google earth
Exit Criteria	The kml file is downloaded from SDC and successfully shows the path of the vehicle
Test Procedures	Step Description	Expected Outcome
1	Login to the SDC website: https://beta-portal-sdc.dot.gov/home	Workstations are displayed at the top right menu.
2	Click the Workstations header, and find a VM listed on the table and then click the Start button in the action column.	The VM is started and the Start button becomes a Stop button.
3	Click the “Launch” button	The Windows login page is displayed
4	Login to Windows and open cyberduck software	The cyberduck connection page is shown
5	Click the “Open Connection” button.	The Open Connection popup is shown.
6	In the top dropdown, select “S3 (Credentials from EC2 Instance Metadata)” and click Connect.	A list of AWS S3 buckets is displayed.
7	Locate and expand “prod.sdc.dot.gov.data-lake.standardized-data”	A list of folders is displayed.
8	Find the “oss4its” folder, and expand this folder.	A list of folders, including BSM, folder is displayed.
9	Expand the BSM folder, and navigate to recorded date time 2021/mm/dd/hh folder	A list of BSM JSON files is displayed.
10	Choose one file and right click on the file name.	A list of options are shown.
11	Click Download to download the file with BSM data.	The file download begins.
12	After the download is complete, navigate to the download folder and open the file to verify it contains BSM data.	The downloaded file contains BSM data.
13	Open Anaconda Prompt.	An Anaconda Promp terminal is displayed.
14	Navigate to where the BSM_to_KML.py script is. Run the BSM_to_KML.py script with below command: “python BSM_to_KML.py <output filename> <bucket name> <bucket folder> <start time> <end time> (eg. python BSM_to_KML.py bsm_output.kml prod.sdc.dot.gov.data-lake.standardized-data /oss4its/BSM/ 2021/08/17/05 2021/08/17/06)	KML files are generate for each BSM json in the S3 bucket between the selected dates.
15	Navigate to and expand the team bucket: prod.sdc.gov.team.oss4its	The team bucked is expanded.
16	Expand and select the export_requests folder	The folder is expanded
17	Click the Upload button and select the output kml file and then click Choose	The file is uploaded
18	Navigate back to the SDC website and click the Datasets header	The datasets are shown
19	Find the uploaded output kml file and click the button under the export column and fill in the required information. NOTE: System administrator approval may be required to export the file	The file is exported and is available to download
20	Click the checkbox next to the output kml file and click the Download button	The file is downloaded
21	Navigate to Google Earth: earth.google.com	Google Earth is shown.
22	Create a new project from the left menu and click the New Project button.	The new project options are shown.
23	Click Import KML file from computer and select the download file.	The KML file is imported into Google Earth and the vehicle’s driven path is shown.

LIST OF ABBREVIATIONS

Acronym	Expanded Form
ACM	American Center for Mobility (Test track)
ADS	automated driving systems
AStuff	AutonomouStuff
CARMA	Cooperative Automation Research for Mobility Applications
CCA	Cooperative Collision Avoidance
CDA	Cooperative Driving Automation
CDL	Commercial Driver’s License
CLC	Cooperative Lane Coordination
CTS	Cooperative Traffic Signals
DDT	Dynamic Driving Task
DOT	Department of Transportation
DMP	Data Management Plan
ECBS	Electronically Controlled Braking Systems
FHWA	Federal Highway Administration
FLETC	Federal Law Enforcement Training Centers
FMCSA	Federal Motor Carriers Safety Administration
GPS	Global Positioning System
GTM	Government Task Manager
HMA	Hot Mix Asphalt
MDOT	Michigan Department of Transportation
NHTSA	National Highway Traffic Safety Agency
ODD	Operational Design Domain
OEDR	Object And Event Detection And Response
PoC	Proof of Concept
POV	Principal Other Vehicle
ROS	Robot Operating System (Software Library)
RSM	Road Safety Message
RSU	Road-Side Unit
RWM	Road Weather Management
SAE	Society of Automotive Engineers
SPAT	Signal Phasing And Timing
STOL	Saxton Transportation Operations Laboratory
SV	Subject Vehicle
TCM	Traffic Control Message
TCR	Traffic Control Request
TRC	Transportation Research Center (Test track)
TIM	Traffic Incident Message
TMS	Transportation Management Service
V2I	Vehicle-to-Infrastructure
V2V	Vehicle-to-Vehicle
VRU	Vulnerable Road User
WZM	Work Zone Management

References

CARMA Confluence: https://usdot-carma.atlassian.net/wiki/spaces/CAR/pages/40861883/CARMA+Outreach+Materials

Work Zone Data Collection Toolset: https://github.com/TonyEnglish/Work_Zone_Data_Collection_Toolset

SAE. “J2735SET: Dedicated Short Range Communications (DSRC) Message Set Dictionary^TM
Set - SAE International.” SAE International, SAE International, 30 Mar. 2016, www.sae.org/standards/content/j2735set_201603.

U.S. Department of Transportation

Federal Highway Administration

Office of Operations

1200 New Jersey Avenue, SE

Washington, DC 20590

Office of Operations website

https://ops.fhwa.dot.gov

August 2021

OSS4ITS Reference Implementation – Test Plan