Testing in the simulator
First, thousands of hours of testing have been flown in the SIMULATOR.
To do this, X-Plane was set up to send data to Xavion that made Xavion think it was in flight based on the Ethernet-based signals that it got from X-Plane. (If you buy Xavion, then you can do this as well… Just get X-Plane 10 and follow the instructions at the instructions link on this site). This allowed engineers from both Vertical Power and X-Avionics to test Xavion for countess flight hours in the lab to perfect the software rapidly and cheaply, without any delays due to aircraft maintenance or weather.
As well as testing by humans, Xavion has been through a strange series of automated tests that have been used to tune its algorithms over about two years. Here is how it works: Before knocking off work each day, Austin would set up X-Plane to send data to Xavion so that Xavion would get data from X-Plane, thinking it was really flying, and put X-Plane on a random self-controlled flight, so the aircraft in X-Plane would go to a random location and fly about at random speeds, headings, and altitudes. Then, after just a few minutes, X-Plane would fail its engine and send the “red-button” command to Xavion. Xavion would then take command of the airplane and fly it down to the best runway within gliding distance, just like it would do in a real airplane in the VP-400, or would command the pilot to do when running on the iPad. Xavion would guide the simulated airplane all the way down to touch-down and roll-out, and the result of the simulated emergency would be recorded by the computer. Each emergency might take 10 to 20 minutes to complete, so after running all through the night, simulating emergency after emergency after emergency, several dozen approaches would be flown.
The next morning, Austin would check on the flights-of-horrors and to see what landings succeeded and what landings failed… and why. Then, he would adjust the algorithms as needed to solve the cases where the landing was not successful. Over the course of about a year, with about 10 hours of simulated emergencies performed each night, thousands of simulated power-off landings have been simulated, handled by Xavion, and analyzed afterwards to see if Xavion could possibly have guided the airplane more safely, with tweaks and improvements made to Xavion as a result of each landing. This worked out to be an evolution of sorts, with countless emergencies tested, and the code updated as a result of each one, bringing it ever-closer to being as good as it could be. As of this writing, Xavion has a very-nearly 100% success rate in the getting the airplane to the ground safely IF THE AIRPLANE IS WITHIN GLIDING DISTANCE OF AN AIRPORT when the engine fails. In other words, with Xavion, the question might not be “Can I maneuver this thing to land if the engine quits?” but may instead be “How high do I need to fly to MAKE it to an airport?”. And, since Xavion easily shows you every airport within gliding range, and what your gliding range is, you instantly know the answer to that question. (One interesting side-note that I learned from these simulations is that a Lancair Evolution can glide close to 80 miles from cruise altitude after an engine failure… and that is within gliding range of at least 2 airports at every single location within the continental United Sates. So, for a Lancair Evolution with Xavion on board in either a VP-400 or an iPad, there is quite literally no place in the Country where an off-airport landing would be needed if the engine failed… SOME sort of runway is nearly assured at every moment of the flight).
Testing in the Cockpit
In flight, Xavion has been most extensively tested by me on various iPads in a Columbia-400 and Lancair Turbine Evolution all over the Nation. As well, I have tested Xavion in a Cessna 172 to test very-low-speed response and GPS reception in metal airplanes. Most of the flight testing, though, has been done in the Columbia-400, where I could test all elements of Xavion in very wide flight profile (from standstill to over 200 knots, and the ground to 25,000 feet). As well, I have tested in winds gusting to at least 25 knots at the surface, with the turbulence that that implies, and winds in excess of 80 knots (so far) at altitude.
The iPad has its own (solid state!) gyros, accelerometers, and (in some models) GPS receiver. I have found that the above internal sensors on the iPad are just barely good enough for emergency or experimental use. The gyros and accelerometers are good enough to track pitch and roll in flight to within about 5 degrees if your maneuvers are smooth and the bank angle is kept to within about 60 degrees or so. If you maneuver more aggressively or exceed 60 degrees in bank, you may see larger errors in pitch and roll display. In non-metal airplanes such as the Columbia-400, the internal GPS on the iPad has been shown to be adequate to receive signal the vast majority of the time, but in metal airplanes such as the Cessna 172, an external GPS of some sort is really needed to get a reliable GPS signal to the iPad.
Because of these limits of the iPad’s internal sensors, external GPSs and gyros are recommended (but not required).
So, for testing in Xavion, I would take off in one of the above aircraft (most often the Columbia-400) and fly some standard flight profiles and emergency power-off descents with both an iPad-2 and an iPad-3 mounted in the cockpit, with various combinations of external GPS and gyros. Unusual attitudes and gentle to aggressive maneuvering would all be tested to compare the attitude of the actual aircraft to the attitude displayed in Xavion, and discrepancies seldom exceeded 5 degree for gentle maneuvering to moderate bank angles. Night and IFR operations were also flown. The engine was pulled to idle countless times (or, in the Columbia-400, pulled to a lower power setting with flaps and speedbrakes deployed to keep the engine warm while simulating a power-off descent), and the test pilot would fly through the hoops presented by Xavion all the way down to the runway threshold, noting, of course, the validity approach presented to that runway by Xavion.
Once the test-card of pre-flight, take-offs, maneuvering, and power-off emergency landings to touch-down were completed, I would start turning off or disconnecting the various external gyros and GPS receivers (to simulate failure of those devices, or you simply not having them), being sure that Xavion would fall back to its internal iPad sensors, and give the best result possible from those sensors. Xavion was only released to the App Store once it was able to present attitude to within about 5 degrees after moderate maneuvering (up to 30 degrees bank and plus or minus 10 degrees pitch) even after every single external sensor was turned off, leaving the iPad running only on its own internal sensors. When an external GPS fails on a WIFI-only iPad, or the internal GPS fails or loses its signal, Xavion goes into a rather interesting deduced-reckoning mode. Even with no GPS signal, the iPad still has its internal GYROS working. With this, the Xavion can tell approximately how much you have TURNED since the GPS signal was lost! In that case, Xavion continues to update your position based NOT on the last known course, but on the last known SPEED… updating the course with the GYROS as you turn the aircraft! Over time, the accuracy of the deduced-reckoning prediction will of course fade, but since the course is updated for turns based on your iPad’s gyros, the deduced-reckoning should track your turns as you change course. This makes the deduced-reckoning in Xavion better than that in other avionics systems that I know of.
Then, once Xavion worked perfectly using only an iPad or iPhone, I added the ability for Xavion to receive ADS-B weather from an iLevil or Sagetech Clarity. With this capability, I flight-tested near thunderstorms and in IFR weather to confirm that the ADS-B weather that Xavion collected was similar to the satellite XM-weather shown on a Garmin-1000, and similar to the weather that I actually observed in flight. This weather included winds aloft, METARs, and NEXRAD precip imagery.In Xavion, NEXRAD precip is displayed on the moving map, winds aloft are displayed on the moving map, and METARs are displayed in plain english for any airport you simply touch on the map. This is the easiest interface I could imagine, and I have found that it matches the XM weather quite well.
In a nutshell, hundreds of hours of testing were done in the X-Plane SIMULATOR to rapidly refine the interface and artificial intelligence and to otherwise develop and tune the product.
Then, thousands of hours of engine-failure testing was done in the SIMULATOR to evolve Xavion’s guidance down to the best runway to land at in the event of engine failure.
Then, I considered Xavion to be ready for the App Store as soon as it could lead a pilot through the:
- pre-flight currency-check,
- cruise altitude selection and planned fuel burn for the flight, and
- weight and balance check,
- recording the take-off to indicate runway use for future take-offs,
- providing standard-six instrument backup,
- providing synthetic vision backup,
- showing hoops down to the (hopefully best) runway to glide down to, guiding the pilot all the way down to the threshold after simulated engine failure, and
- doing all of the above after the failure of every GPS and gyro system the iPad could get input from (using the iPad’s internal sensors only),