Development of software for an information system that points the way using augmented reality technology Desarrollo de software para un sistema de información que señala el camino utilizando tecnología de realidad aumentada

This paper describes the development process for the map directions information system application utilizing augmented reality technology. The first part of this paper discusses issues related to the research objective formulation. The objective and tasks to be solved are determined. Theoretical information and developments in the subject area are discussed. The second part describes the developed application architecture, the choice of a programming language is explained. The third part expressively addresses the development of algorithms and the application itself. The next section contains instructions for end users. And finally, a summary of the concluded work is given and relevant conclusions are made about the obtained results. It is emphasized that the work leads to the creation of an application for route finding and navigation in augmented reality mode. It is noted that this application can be extended and enhanced by developing versions for other mobile operating systems (for example, iOS and Windows Phone).


INTRODUCTION
Mobile devices are becoming more popular with each day, which is not surprising, as nowadays these devices help finding any necessary information or performing complex operations without much effort. The most important factors making smartphones more popular than PCs and tablets are their portability and relatively low cost. The objective was to develop a high-quality handy application with elements of augmented reality, providing users the capabilities to create convenient routes. At the moment, one of the most promising areas of IT development is augmented realityan environment where the real world is augmented with digital data, perceived as real-world elements (Tsvetkov, 2017;Yakovlev & Pustov,;Yakovlev & Pustov, 2013). When the user deviates from the route, the application helps him to find the direction towards the selected location using a direction pointer in augmented reality (AR). Therefore, this scientific paper is aimed at developing the program code for AR Navigator mobile application, which main advantages are state-of-art technologies and modern-day relevance.

Theoretical information
The developer of a mobile application is faced with the task of creating a route to a destination and incorporating elements of augmented reality. The application should have the following features: • destination object selection; • creation of a path to the destination object from the current coordinates; • splitting the path into coordinates and mapping different types of pointers to them; • drawing pointers in augmented reality. System requirements: smartphone running Android 4.3 operating system (OS) or later, having a camera, GPS, compass, accelerometer, Internet connection; Android Studio 3.1 development environment, Java 8 programming language. Android Studio development environment is cross-platform, allowing to develop Android applications on the following operating systems: • Microsoft Windows 10 (32-or 64-bit); • Mac OS X 10.10 or later; • Ubuntu 14.04 LTS (32-or 64-bit) or later. The input data for the application is data obtained via an Internet connection and smartphone sensors, namely: GPS, orientation sensor, compass accelerometer. In the absence of an accelerometer, the application can receive data from other sensors, such as gyroscope or magnetic field sensor. The choice of input data source is based on the availability of higher priority sources. If there's no access to a source or data available, the system automatically requests data from a lower priority source. The information is provided to users in the form of graphic elements such as labels, arrows, shapes, etc. The data in the displayed items depends on the set of actions performed by the user. The developed mobile application is intended to display the current location and create a route to the location selected on the map, using direction pointers in augmented reality. The choice of input data source is based on the availability of higher priority sources; if there's no access to a source or data available, the system automatically requests data from a lower priority source. The information is provided to users in the form of graphic elements such as labels, arrows, shapes, etc. The data in the displayed items depends on the set of actions performed by the user. The developed mobile application is intended to display the current location and create a route to the location selected on the map, using direction pointers in augmented reality. At the moment, Android OS is the most popular OS in the world (Figure 1) (https://gs.statcounter.com). Such popularity arises from Android OS simplicity, functionality, portability and supported device types. According to the official Android OS website, currently, Android OS works on such devices as smartphones, smart-watches, tablets, smart-TVs, and is also successfully integrated into cars (https://www.android.com). Many books, documents, articles and internet resources are devoted to Android OS application development. All the necessary sources can be found in both English and Russian. Literature sources (Caudell & Mizell, 1992;Milgram & Kishino, 1994;Azuma, 1997;Boychenko & Lezhankin, 2010;Ivanova, 2018) and (Yegorov, 2019;Kuraev, 2019;Chirkin & Pimonov, 2019;Honcharova, 2019;Pavlova et al., 2020) related to the subject of this paper have been reviewed. The book by Goloshchapov A.L. (2011) addresses the following aspects of application development: application development for mobile devices running Android OS, basic information about the Android platform, software necessary for the development of Android applications. Practical course by Reto Mayer (2011) is devoted to software development based on Android SDK 2.0 (software development kit). The examples presented in the book reflect real programming tasks based on the studied theoretical material. Android for Programmers book (Deitel et al., 2012) contains massive amount of theory. The text of each subsection contains illustrative code samples of sixteen widely used real-world Android applications. The official Android website for developers has a dedicated resource (http://developer.android.com) with various programming courses, API guides (APIapplication programming interface, a set of predefined classes, procedures, functions, structures and constants provided by the application for use in external software products (https://en.wikipedia.org)), quick references, tools for developers and application code samples. In 2014, Google provided its vision of Android OS software and application design philosophy with the Material Design framework (https://ru.wikipedia.org). Material Design implies that applications open and collapse like cards with shadows effects (https://design.google.com). The official JetBrains website has a dedicated resource providing useful plugins for Android Studio (https://plugins.jetbrains.com), in particular, for building UML diagrams. One of the main unique aspects of this paper is utilization of the mobile device's integral sensors. Alexander Klimov's website (http://developer.alexanderklimov.ru) contains information about all sensor types used in mobile devices, as well as examples of their use. Google Maps Android API is used to enable the map view, calculation of distances and areas, as well as finding directions using spherical geometry for the application. More details on this interface can be found on the official Google Maps API website (https://developers.google.com; https://developers.google.com), containing the necessary tutorials on this subject. To implement the augmented reality feature during the application development, it is necessary to solve a number of geodesic problems, requiring understanding of the geodetic theory, in particular, a concept of azimuth (https://en.wikipedia.org). For the initial stage of the mobile application development, the paper considers certain issues, among which, for example, is the choice of application design pattern. Existing design patterns and their implementation methods are described in the books "Design Patterns: Object-Oriented Design Techniques." (Gamma et al., 2016) and "Elemental Design Patterns" (McC Smith, 2013). Today, the mobile application market is one of the most highly-demanded. According to the We Are Social analytical company statistics, as of April 2020, smartphones remain the main tool for accessing the Internet, and generate more web traffic than all other devices combined (Figure 2) (https://wearesocial.com). Android OS is the most popular OS in the world, which implies a high market demand on applications for this platform. Also, application development for this OS has a low entry threshold, as it does not require knowledge of domain-specific programming languages, and has low requirements for the computer's hardware and OS, where the development environment will be installed. For example, application development for iOS mobile OS, as well as MacOS X for Apple computers, requires knowledge of Swift programming language, and application development for Windows Phone requires the developer to have knowledge of C# and Windows OS specifics. Android applications consist of bytecode running on the ART virtual machine. Android OS offers two main approaches of application development: using Java programming language SDK, or C/C ++ programming language NDK. SDK (Software Development Kit) is a set of classes and libraries for Java programming language allowing to utilize the capabilities of Android OS. NDK (Android Native Development Kit) is a package of tools and libraries allowing to develop applications in C/C++. NDK is recommended for developing code sections, where performance is crucial. Assuming that the developed application does not require high performance, the best solution would be to use Java SDK. To make the development process more comfortable, certain development environments (IDE) are used. Two major IDEs are currently used for Android development -Eclipse with ADT plugin (Android Development Tools) and Android Studio. For a long time, Eclipse ADT plugin development was held by Google, but after the release of Android Studio IDE in August 2015, the company stopped supporting this plugin, which means that Eclipse can no longer provide full support for Android OS application development and debugging. Therefore, using Android Studio IDE is a natural solution in this situation. Design patterns are an essential part of any software. They help keep the code clean, make it scalable and testable. Many design patterns are used in Android development, and we will consider such patterns as: MVVM, MVC, MVP. MVVM (Model-View-View Model). This approach allows mapping user interface elements to properties and events of the View model. The idea behind MVVM is that each layer of this pattern is not aware about the existence of another layers (https://en.wikipedia.org). MVVM is designed for team-based application development, i.e. while one team member is working on the layout and screen styles, the other can simultaneously code the data reception and processing logic ( Figure 3). MVC design pattern (Model-View-Controller) has been created to separate business logic from user interface (https://ru.wikipedia.org). The idea behind this design pattern is that the user interacts with the controller, the controller requests data from the model and sends it into View, which is displayed to the user ( Figure 4).

Figure 4: MVC design pattern diagram
One of the most efficient design patterns is MVP (Model-View-Presenter) (https://ru.wikipedia.org), since it allows separating user interface (View) from the data itself (Model), while providing interaction between those via a "middlemаn" (Presenter), which simplifies Unit test creation and application scaling ( Figure 5). This development approach is relatively new, but is rapidly gaining popularity due to its simplicity and efficiency in solving many problems. This design pattern will also be added to the new version of the Java language. The essence of this approach is to use asynchronous data streams. This approach ensures that applications are more resilient to network errors, while simplifying the processing of various events. Therefore, this programming approach has been used in order to increase the application stability. Before writing this paper, we set a general objective to develop an application that will quickly and efficiently provide users with real-time information about their location and create a route to their destination of travelling. The developed mobile application requires Internet connection via Wi-Fi or GPS. Let us considering the following particular tasks that are solved in the software development process:

User action
Uses methods Change events Changes application architectural design; receiving data from sensors and Google services; map loading; filtering and displaying calculation results; route creation; camera utilization and event display. After the application is launched, a map view is displayed including the user's current location. Data is received via the Internet connection. In order to include the map in our application, we need to register on the Google Maps website and get the relevant API key. The application's main screen with a map includes a button (map icon) for route creation. After being touched, 5 markers connected by a line are added to the screen. The markers can be repositioned. The screen also includes buttons to delete the route and switch to the camera view. Further, the user can switch to the camera view by clicking on the button with a camera icon. Switching to another screen does not affect the data about the created route. Data is retrieved from smartphone sensors and via the Internet connection, after which the data is processed and displayed in an information window, as well as in a form of images or pointers. To get the pointer indicate the direction, the angle between the user's coordinates and the map marker shall be calculated. For this, azimuth shall be found, as well as the distance between the user and the map marker. The user sees an arrow on the screen indicating the direction to the first marker on his route. When the user is in the target range, taking into account permissible deviations, the augmented reality object is displayed on the screen, after which the application creates the route to the next point, etc. After the entire route is completed, application switches to the map view, allowing to create a new route.

Software solution architecture
Android application development does not require the developer to use specific hardware and/or software platform, which simplifies the software development process. We have used Google's Android Studio Integrated Development Environment (IDE), which includes all the necessary tools for application development, debugging, and testing. Java 6 is the minimum requirement for working with this IDE and compiling applications, although Java 7 or later is recommended. Android Studio has rich functionality that helps to accelerate application development (https://developer.android.com). This IDE includes a number of necessary tools for application development (Mikheev et al., 2016). To be able to use our application, the user should have a smartphone with a set of necessary sensors, and clean up the smartphone memory cache occasionally to avoid errors inside the application. A review of relevant programming languages is given below.

Justifying programming language choice
Swift is a fast and efficient programming language with real-time response that can be easily inserted into a fully-functional Objective-C code. It eliminates a large layer of common software errors by leveraging modern software design patterns:  Variables are always initialized before they are used.  Array indices are always checked for "out-of-bounds" errors.  Integers are checked for overflow.  Optional guarantee that nil values will be explicitly processed.  Automatic memory management.  Error handling enables controlled recovery from unexpected errors.
C# is an object-oriented language with strong class typing, allowing developers to create safe and reliable applications of all kinds, based on .NET Framework. In addition to mobile application development, C# can also be used to create Windows client applications, XML web services, distributed components, client-server applications, and database applications. Java is an object-oriented programming language. Initially, the new programming language was called Oak (by James Gosling) and was developed for consumer electronics, but was later renamed to Java and began to be used for writing applets, applications, and server software . Java include 3 main technology families: J2EE -Java Enterprise Edition, for creating industrial-grade software. Figure 6 shows J2EE structure. J2SE -Java Standard Edition, for creating desktop applications, primarily. J2ME -Java Micro Edition, for embedding into devices with limited computing power, including mobile phones, PDAs, embedded systems. Java with a set of relevant API libraries is considered the main programming language for Android OS. Android Developers website is regarded the best resource for Android system development guides (Goloshchapov, 2011). Regarding the development IDE -Google has developed a number of additional features for the IntelliJ IDEA IDE and eventually released an IDE called Android Studio. In addition, Java Development Kit (JDK) version 5 or later is required. Out of the abovementioned options, Java programming language is the most suitable for the execution of objectives, as it includes an extensive library of classes, frameworks, etc.

Choice of software development environment
Since we have chosen Java as the programming language, application development will be carried out in Android Studio IDE, being the most relevant for Android development (Martyshkin & Martens-Atyushev, 2019). Each project in Android Studio contains one or more modules with source code files and resource files. The following module types are possible: -Android application modules.
-Library modules.  (Figure 7). All assembly files are visible at the top level under Gradle Scripts, and each application module contains the following folders: manifestscontains AndroidManifest.xml file; javacontains Java source code files, including JUnit test code; rescontains all non-code assets, such as XML layouts, user interface strings and bitmaps. - The data structure development is one of the most important tasks at the initial stage of development. Often, the data, in particular, determines the possible functionality of the developed application and its operation principles. MVP design pattern used in this paper allows to separate the data from their presentation, which makes it possible to store data in any convenient form and change it without impairing the main application logic. Input comes from mobile device sensors and Google services. SensorManager class is used to work with hardware sensors. Regardless of how many sensors will be used in the application, all sensors work independently of each other and data from them are easy to acquire. However, working with sensors significantly increase energy consumption, and that fact shall be taken into account. Typically, real devices are used to test similar applications that utilize hardware sensors, and it is best to ensure that the smartphone have these sensors. This can be done by reviewing the smartphone specifications or by coding a small application to display readings from all sensors. After the data is received, it is presented in the form of a data stream, which contains class structures. Data composition and structure related to route creation with augmented reality are given in Tables 1-5.     All data received from the services are stored in the local storage, which allows to retrieve data from the local storage and continue working with it when Internet connection is lost or in case of a network error. The data from sensors do not require long-term storage, as it is constantly changing.

Algorithm development
After launching the application, the user needs to allow detection of his phone location, after which his location is detected and the map view is centred at it. If the location cannot be detected, then data on the user's last location will be retrieved from storage, in other case a message will be displayed on the screen with a hint. A route can be created after that, which can be modified or deleted as well. Further, user can touch the button with a camera icon to switch to the camera view. If the user has committed some wrong action, then a message will be displayed with a hint at the bottom of the screen. More details about such messages are provided further in the text. The user can freely switch between the camera and map views, however, if the user has not travelled the entire route and returned to the map view, he will have to create the route again, since the route is not saved. Screen navigation diagram is shown in Figure 8.

Figure 8: Screen navigation diagram
When the user switches from the map view to the camera view, the route data is passed using Intent method, after which the received data is recorded in predefined variables and used within the code. Camera and map classes use data obtained from sensors, which is done by running classes created to receive readings from sensors. To initialize the classes, start () method is called and an interface is implemented to pass the data. Immediately after switching to the map view, data retrieved from the Internet and device sensors is processed, after which the screen is loaded with the camera view displayed and the direction pointer to the first route marker. The pointer is created by mapping the orientation sensor to the pointer, after which the azimuth is calculated by the formula (1). The azimuth value is required in order to find the angle between the user and marker on the view.
To be able to properly display the augmented reality object, both the azimuth and distance values shall be known. The distance is calculated by the formula (2).
= √Δx 2 + Δy 2 , (2) To make the application more user friendly, the code includes constants for permissible azimuth and distance deviations, which are used to make the object easier to display on the camera view. When the user is in the target range, taking into account permissible deviations, the display condition is satisfied and the augmented reality object is displayed on the screen, after which the application receives coordinates for the next marker and performs the same operations with it. After the entire route is completed, application switches to the map view, allowing to create a new route.

Application development
The developed AR Navigator mobile application is intended to show directions to the destination object using a map and camera with augmented reality. The application shall be written in the Java programming language and have the functionality specified in the beginning of this paper. The application starts with MapActivity class. When this class is initialized, onCreate() method is called, which initializes the map, connects to Google services, requests access to the current location and triggers the map marker handling method (Figure 9). To add a map to the fragment, createMapView() method is used, which connects to Google services and displays the map in case of a successful connection to the Internet (Figure 10). An error message is displayed if the map could not be created. After the map is added, route creation can be initiated. To do this, the application needs to know the user's coordinates and add markers connected by a line, get the latitude and longitude of the device and assign their values. CameraViewActivity class implements the main idea of the application. When switching to this Activity, OnCreate() method is triggered (Figure 11), after which the following chain of events "Map creation error" occurs: route data is received from MapActivity class, MyCurrentAzimuth and MyCurrentLocation class constructors are called and their start() methods executed, screen elements and camera view are initialized, an augmented reality object instance is created.  Objects package classes describe the data structures received by the application from the services and sensors they interact with: -ArObjectdescribes augmented reality object data; -MyCurrentAzimuthdetermines the azimuth; -MyCurrentLocationdetects the device location; onAzimuthChangedListener interface describes a method for azimuth calculation.
request method; passed attributes; onLocationChangedListener interface describes a method for device coordinates handling and includes: request method; passed attributes; MyCurrentLocation class includes buildGoogleApiClient method, which creates ApiClient that uses .addApi method to call the location discovery. A request is created and the interval for request sending is set (Figure 13). // getting route points data from the map fragment Figure 13: start() method of MyCurrentLocation class start() method of MyCurrentAzimuth class connects to a sensor and registers a sensor listener with a specified refresh rate. Data is collected from the accelerometer, gyroscope and magnetic field sensor. After being called, the method is triggered every time the phone moves in space ( Figure 14). To process and output data from sensors, SensorEventListener interface with the two methods onSensorChanged() and onAccuracyChanged() are implemented. Data is received via an asynchronous stream ( Figure 15). onSensorChanged() method is called when a new sensor event occurs, and a device rotation matrix is retrieved.  Data processing and displaying on the screen occurs in Activity classes, after which the collected data is passed to the required class by calling methods from the data collection classes. The following is required to ensure correct functioning of the application: smartphone with Internet connection and GPS; relevant mobile device sensors.
To use the application, a real device running Android 4.3 or later, or a virtual machine with similar specifications is needed. If there is no real device, the application can be installed and tested on an Android OS emulator. For this, install the basic SDK toolkit from the website (Goloshchapov, 2011). Unpack the downloaded archive into C:\Program Files\android-sdk folder. Run SDK Manager.exe file from this folder. The window will be displayed as shown in Figure 17.     To do this, run the command line, change to C:\Program Files\android-sdk\platform-tools folder, and execute the following command: adb install -r <file_name>, where <file_name> shall be replaced with the full path to AugmentedRealityNavigator.apk file. The -r parameter indicates that if the application was installed on the device before, it will be deleted first and then installed again. The application can be launched by selecting the application icon from the list of installed applications.

User instructions
AR Navigator mobile application is a map directions information system utilizing augmented reality technology. Before starting, the device shall be connected to the Internet and with GPS enabled, to enable the map view and obtain device location data. In case there is a problem with the map and it is not drawn, a relevant message will be displayed (Message No. 1, Table 6). After launching, the main application screen is displayed (Figure 22, a). This screen contains a built-in Google map view, as well as the application control buttons. By touching My Location button in the upper right corner, the user's location data is downloaded, and the map view is centred at the current location. If the data could not be loaded, the user's last known location will be displayed. The user's location is displayed on the map as a small blue dot (Figure 22, b). The lower part of the screen has 3 essential buttons, namely: create a route, delete a route, switch to augmented reality mode. By touching Create Route button with a map icon, 5 markers are added to the map near the user's location, connected by a line. These markers have numbers and can be moved (Figure 22, c). If the route has already been created, a relevant message will be displayed on the screen (Message No. 5, Table 6). An augmented reality pointer is mapped to each marker. If the user's location has not been determined, a message will be displayed on the screen (Message No. 2, Table 6). By touching Delete Route button with an eraser icon, a check is performed to find any connected markers on the map, which are deleted along with the line connecting them if found. If no route is found on the map, a relevant message is displayed (Message No. 3, Table 6). By touching button Switch to Augmented Reality Mode with a camera icon, the application switches to a new window with a camera view (Figure 23, a). Before touching this button, a route shall be created first. If no route exists, a relevant message will be displayed on the screen (Message No. 4, Table 6). After switching to Augmented Reality Mode, a circle with an arrow will appear on the screen, showing the direction to the first of five points on the route. The bottom of the screen displays how many route points the user have travelled, the distance to the next point and a button to switch to the map view. When the user arrives at the destination, the marker image appears on the screen in an augmented reality mode (Figure 23 After the augmented reality object is displayed, the bottom screen information is updated and the route to the next point is created, after which the AR object remains on the screen for a while, and then disappears. When the user arrives at the last point, the augmented reality loop will end and the application will automatically switch to the map view, allowing the user to create a new route. Below are minimum system requirements for the application to work correctly: -Smartphone running Android 4.3 or later; -CPU clock frequencyat least 400 MHz; -RAM sizeat least 512 MB; free internal spaceat least 100 MB; -Internet connection; active GPS; camera; compass, accelerometer, gyroscope, magnetic field sensor. The main input data for the application is data received via the Internet connection and GPS, as well as readings collected from sensors. To use the proposed application, the user should have a basic knowledge of operating cell phones with a touch screen. All interaction with the device is done via a touch screen. The approaches used in the application development remove a number of operations required for application maintenance. But in order to ensure stable operation, it is recommended to clean RAM and internal memory cache of the device, check its operation, as well as connection to the Internet and GPS status.
The application was developed and tested using Xiaomi Redmi 4 with the following specifications: -Android OS 6.0.1; -CPU clock frequency -2260 MHz; -RAM size -3 GB; free internal space -32 GB.

CONCLUSIONS
Based on the results of the research, this paper presents a developed system, implementing the functions specified in the research objective. The application allows using mobile devices for quick and efficient route creation, and includes augmented reality elements. The pros of the system include utilization of promising mobile technologies that are gaining widespread popularity, code succinctness, simplicity of use on a regular mobile phone, and an intuitive user interface. The fact that the system is running Android OS installed on a mobile device can be seen as an advantage as well. Not every mobile phone is based on this OS, but, namely, this platform is currently the most popular one. The advantage of Android OS is that phones of different price categories are produced to run under this OS. The distribution of the proposed application and its implementation are also quite easy, since Android OS developers have provided a special Internet resource for distribution of both free and paid applications. This application can be extended and enhanced by developing versions for other mobile operating systems (for example, iOS and Windows Phone).

ACKNOWLEDGMENT
The article is published with the support of the scholarship from the President of Russian Federation to young scientists and graduate students in 2018-2020 (SP-68.2018.5).