Introducing an intelligent multi-level retrieval method for mineral resource potential evaluation result data

Li Chaokui; Liu Mingxi; Guo Ruirong; Zhao Yanan; Yang Wentao; Zhang Xinchang

doi:10.1515/geo-2022-0504

Article Open Access

Introducing an intelligent multi-level retrieval method for mineral resource potential evaluation result data

Li Chaokui , Liu Mingxi , Guo Ruirong , Zhao Yanan , Yang Wentao and Zhang Xinchang

Published/Copyright: December 31, 2023

Published by

Become an author with De Gruyter Brill

Submit Manuscript Author Information

From the journal Open Geosciences Volume 15 Issue 1

Abstract

The geological data of the mineral resource potential evaluation results (MRPERs) are diverse and extremely large; efficiently retrieving data remains a challenging problem. In this work, a new way of using the Hadoop platform is proposed. The Hadoop distributed file system is used to store the massive data and construct the data storage model of geological and mineral resources. Using a distributed Hadoop database (HBase) that supports the fast query of a single record, it manages its metadata and retrieves the data of MRPERs quickly. At the same time, a multi-level index directory is designed to support the non-main key query on the HBase. This overcomes the shortcoming that the HBase only supports the simple index based on the main key and realizes the intelligent, efficient retrieval of MRPERs. The validity and feasibility of the proposed method are further verified by experiments using the MRPER data in the Institute of Mineral Resources, Chinese Academy of Geological Sciences.

Keywords: mineral resources; potential evaluation; results data; Hadoop platform; HBase multi-level index method

1 Introduction

The era of geological big data has arrived [1], driven by the on-going development of the “Digital Earth” and “Digital Territory” fields. In China, for example, two large-scale geological data collection and evaluation projects have been conducted, which have established an extremely large collection of geological data. The first (1979–1985) has identified the main metallogenic belts; the second (1992–1996) has evaluated the mineral resource potential in these belts. The data are obtained through mineral resource exploration and summaries of geological survey results, which can be used to scientifically evaluate the potential of unknown mineral deposits. The massive data consist of a wide variety of data that include both structured and unstructured data. This data collection is of great significance for guiding prospecting activities. While the management of massive geological data has largely been realized, it is very important to obtain the required data efficiently from the data of the national mineral resource potential evaluation results (MRPERs). The real-time, efficient retrieval of data remains a challenging problem in geological big data research.

Valuable technologies, including the Hadoop framework, are available to practitioners and researchers working on big data problems; Hadoop is adopted in this research. Hadoop is an open source framework for distributed systems developed by the Apache Foundation. The core of the Hadoop framework is Hadoop Distributed File System (HDFS) and MapReduce [2]. HDFS is mainly used for distributed storage of large-scale datasets. MapReduce enables parallel and efficient processing of distributed large-scale datasets [3]. Hadoop is a popular Share-nothing-based distributed processing system, with the advantages of high efficiency in data processing and easy expansion [4]. As the underlying infrastructure, the HDFS provides high reliability and high-performance storage services for cloud computing, which can be used to solve the storage problem of geological big data. Since a simple HDFS is not suitable for low-latency access applications, a distributed Hadoop database (HBase) that supports efficient retrieval is used to meet its retrieval requirements. Hu and Wu [5] proposed a Hadoop-based trajectory data preprocessing method. They used distributed storage, small file merging, MapReduce programs, etc. to build a data preprocessing environment, complete the data cleaning, and finally realize the trajectory correction. Khalil and Hamad [6] used a framework to improve the performance of a query and reduce the response time called the Hadoop. In view of the data of the MRPER, a column storage form is designed to store the evaluation metadata of the MRPER in the HBase, which is convenient for efficient retrieval. At the same time, the HBase multi-level index directory is designed. Experiments demonstrate that the proposed multi-level retrieval method is feasible and efficient for very large data collections.

The remainder of this article is organized as follows: Section 2 details the background and related work, Section 3 details the intelligent data retrieval for the MRPER, Section 4 describes the experiments performed, and Section 5 concludes the article.

2 Background and related work

2.1 Applications of NoSQL technologies in big data retrieval

With the rapid development of processing technology for extremely large datasets, i.e., big data, the NoSQL technology has become a hot research field. Traditional geospatial data retrieval is usually focused on the character or semantic features of search terms, and the matching degree between the search terms and geospatial data is studied. In terms of character feature similarity, early domestic and foreign scholars have proposed retrieval algorithms based on Hash, Trie index tree, and double-word Hash by referring to dictionary query methods. Unlike traditional databases, the NoSQL database gives up many rules in the “12 Codd’s rules” that the relational database management systems follow and makes innovations in storage and application. It is difficult to define its retrieval methods because they themselves lack a unified technical solution [7]. However, three recognized categories of indexing with NoSQL databases are as follows:

Single-level index: NoSQL supports a “healthy index” or “primary key index” to achieve single-level data retrieval, and its implementation mechanism is similar to B+Tree adopted by relational databases.
Two-level index: NoSQL has the characteristics of weak atomicity, consistency, isolation, persistence, and open source and distributed characteristics, so it is difficult to achieve two-level data retrieval under the conditions of low redundancy and high consistency of data. At present, a limited number of methods, such as Cassandra, support two-level data retrieval.
Full-text index: NoSQL-customized support for full-text data retrieval can be developed using existing toolkits, such as Lucene, to meet actual needs.

Various NoSQL database products are under continuous development including the Hadoop open-source framework for distributed cluster-based processing and analysis of big data sets. The framework includes the HBase, HDFS, and MapReduce. Hadoop can be deployed on many inexpensive machines. This framework is widely adopted in scholarly investigations on a variety of big data storage, management, and retrieval topics. HBase is a column-oriented database management system. Unlike relational databases, HBase does not have strict form rules. It can include both vector graphic and text-type data; the data records of the storage table may also contain columns of different sizes. HBase runs on top of the HDFS, which has a high fault tolerance feature that allows users to deploy distributed systems on inexpensive physical machines to provide data storage services [8]. MapReduce [9] is the framework of processing distribution that implements the model of the same name. Zookeeper is a tool for monitoring distributed services that determines how to operate a machine [10]. Recently, the research community has used HBase to build and store index tables and index data, in addition to achieving the fast query and retrieval of web pages by the distributed inverted index method [11]. The R-tree has been written into an HBase using the application to achieve an HBase multi-conditional query [12]. By designing a consistent hash distributed memory caching mechanism, an HBase has been implemented based on non-key retrieval and range-based retrieval [13]. The HDFS has been used to save index files and achieve distributed queries on HDFS files [8]. The Lucene information retrieval library has been used to achieve an HBase full-text retrieval function [14].

2.2 Data acquisition and management of the MRPER

2.2.1 Surveys to establish the MRPER

The data of the national MRPER include many topics, a wide range of sources, and a large coverage area. The data have been acquired by the National Geological Institutions. The total amount of data has reached 13T, including more than 20 mine types, 47 projects in total, and 9 types of geological data [15]. The survey data include MapGIS format point file, line file, graphic file, picture data, metadata file, description file, and table data. The vector data are derived from the national MRPER using the national unified system library and the MapGIS 67 system.

2.2.2 Data management models of the MRPER

Since the 1960s, the US has actively pursued research to establish geological databases and has invested substantial resources to accomplish this. The other developed countries are following. According to the geological conditions of each country, a database of gravity, magnetic force, hydrology, and mineral resource distributions in each country has been established. These efforts have used diverse technologies including grid computing, massive graphic library management, and multi-type spatial data integration [16]. China’s geological data management is also moving forward steadily. Its management system is the result of a comprehensive application of geographic information system (GIS), database technology, and network technology [17]. Since 2007, the Ministry of Land and Resources has carried out the national MRPER project, and the establishment of the mineral resource potential evaluation information management system has become a hot research topic for domestic scholars. Ye et al. [18] have combined two development methods, MapGIS K9 and GIS, to realize the data management system of the MRPER in the Tibet area. Based on Oracle 11G and the MapGIS geographic database, Zuo [19] has established a data model of the MRPER. Zhu et al. [20] have completed the data integration of Jiangsu's MRPER by using MapGIS, SOL Server 2008, and GeoMAG software.

3 Intelligent data retrieval for the MRPER

The MRPER data collection is extremely large and has a complex structure. The collection includes vector space data, remote sensing image data, text, and so on; it is a mix of structured and, increasingly, unstructured data. Therefore, building an intelligent retrieval system of geological big data that is efficient and stable is of great importance. The proposed intelligent retrieval model of national MRPER mainly includes three functional components: the data storage module, the parallel computing module, and the intelligent retrieval module. The structural view of the architecture is illustrated in Figure 1.

Figure 1

Intelligent retrieval architecture.

3.1 Data storage module

Based on Hadoop, a data storage model of MRPER is constructed in this article (Figure 2).

Figure 2

Data storage model of the MRPER.

The underlying physical storage interface is unified through the HDFS and its related technologies; the storage devices in the system are unified into a resource by virtualization to achieve multiple DataNode coordination. The data of the national MRPER are stored on the HDFS, and the index files are stored in the HBase column storage table by parallel computing. This model provides an efficient data storage solution.

3.2 Parallel computing module

As a parallel programming model [21], the MapReduce component is used for the parallel computation of big data. Its working process is mainly divided into a map phase and a reduce phase. Based on MapReduce, we use the TableOutputFormat method provided by the HBase to achieve the index data of the MRPER; these are efficiently imported into HBase. In the map phase, MapReduce divides the national MRPER metadata on the HDFS into fixed size partitions. Subsequently, each slice is decomposed into the form of a key, value pair, which is represented as <k1, v1>. Hadoop will create a map task for each split to execute the user-defined map function. At this time, <k1, v1> is used as the input key-value pair to obtain the intermediate value: here represented as <k2, v2>. Then sort by k2 and combine the values corresponding to k2 to form a <k2, list(v2)> tuple. Finally, <k2, list (v2)> is grouped according to the range of key values. In the reduce phase, the data received from the different maps are integrated and sorted. The process calls the reduce function written by the user, handles the input <k2, list (v2)> tuple, and produces <k3, v3>, which is the output to the HBase. In this article, a 50,000 data volume collection (text data) stored in HDFS is imported into the HBase through the MapReduce parallel algorithm to form the program execution of the index table. The task execution is shown in Figure 3.

Figure 3

MapReduce task execution: 50,000 text data sample.

3.3 Intelligent retrieval module

3.3.1 Design of the data index table for the evaluation of the MRPER

The HBase stores the index data of the MRPER in the form of a column storage table, so as to form a sparse and multi-dimensional sorting mapping table [22,23]. In the HBase, the table name is used as a uniquely identified table; the line keyword is used as the primary key to uniquely identify a row of data. When querying the row data in the HBase, three forms can be done: a single row of keywords, a given health range, and a full table scan. Under the column family, the number is not strictly regulated and can be modified according to the needs of the users, so as to ensure the flexibility of the HBase storage. Unlike relational databases, the HBase has no strict form rules, including both vector graphic data and text type data, and the data records of the storage table may also contain columns of different sizes. Therefore, this article designs the logical model and physical model of the data index table of the national MRPER.

Logical models (mappings of ordered mappings)

The HBase uses the coordinate system to query the data in the unit: (rowkey, column, column qualifier, timestamp). The logical model of the data index table of the MRPER is designed as shown in Figure 4.

Figure 4

The logical model of the data index table for the MRPER.

The concept of the Figure 4 model is described using the co-ordinates from the inside to the outside. It starts with Key and the data is a unit maps to a value, then the column qualifier is key, and the unit map is a column family map. Finally, the column is the Key, and the column family is set up to build the table map for the Value.

Physical model (column oriented)

The column family in the HBase contains columns, and each column has its own HFile sets on the disk. This forms the physical isolation, allowing data to be managed separately at the HFile level, and the physical model of the index data of the MRPER is stored in the HFile, as shown in Figure 5.

Figure 5

The physical model of data index table for the MRPER.

The MRPER data index table has no empty records in the physical model of the data index table. If there is empty data, the HBase will not store the data in the column. Hence, the HBase column storage table is column oriented, and the same column family needs physical storage in one row. The data stored in the HFile in 55290602072625 rowkey are complete. If the data have more than one column family at the same time, then each column family uses its own HFile. This means that when reading data from the HBase, it is not necessary to read all the data in a row, only the part of the column family used in the data needs to be read in, which provides efficient data storage and a fast read.

3.3.2 HBase multi-index method

The HBase only supports the fast retrieval based on a primary key and does not support query based on the data of non-primary key, which greatly restricts its application. Based on MRPER data, a multi-index based on metadata management table is proposed, in which the primary or secondary index retrieval is as shown in Figures 6 and 7.

Figure 6

The MRPER of the multi-level index metadata (first level).

Figure 7

The MRPER of the multi-level index metadata (second level).

In order to avoid problems caused by redundant combinations of RowKey metadata and enable other cached data to use memory more efficiently, we design the RowKey index table to be shorter. The retrieval process is as follows:

First of all, if the client wants to query the value of cf:address of cf:map = “552906 Gravity Data Application”, according to the HBase primary key query rules, a full table scan of the component is required to obtain the required data. However, through the multi-index method, the Map table can be retrieved, and the level of the RowKey table from the Map table is found. Then, the data in accordance with the conditions set from the column corresponding to the cluster are obtained, and the search program extracts the condition value from this small set. The retrieved value value is assigned to the Component table and located at the corresponding cf:address, thereby further calling the mineral resource port capacity evaluation result data stored in HDFS.

Because the geology and MRPER with hierarchical relations are more complex, the multi-index method is used with the designed mineral table to represent the hierarchical relationships between the various attributes. According to cf:mineral = 5529 “retrieval hypothesis,” which contains witherite maps, only part of the component table corresponds to the primary key. At this time, the mineral table and the corresponding values through the Rowkey “5529 witherite” are retrieved. Then, the value is used as the new Rowkey to retrieve the map table, which contains the map information. This proposed approach realizes the intelligent retrieval of MRPER data.

4 Experiment

4.1 HBase contrast test: before and after optimization

4.1.1 Hardware environment

The experimental environment has seven virtual machines to build clusters, of which three are for DataNodes, one is for NameNode, and three are for Zookeeper and HBase. Table 1 shows the hardware environment.

Table 1

Cluster configuration

Component	Configuration
Hadoop Version	Hadoop2.7.2
Hbase Version	Hbase1.1.3
Zookeeper Version	Zookeeper3.4.7
Operating System	Red Hat Enterprise Linux Server release
	6.7 (Santiago)
Linux Kernel Version	2.6.32–573.e16.x86_64
JDK Version	1.7.0_91
Network Bandwidth	100 MB
NameNode	Eight-core 2.40 GHz CPU, 16 G RAM, 600 G ROM
DataNode	Eight-core 2.40 GHz CPU, 8 G RAM, 600 G ROM

4.1.2 HBase before and after optimization of retrieval experiments

In order to compare the proposed multi-level intelligent retrieval with the traditional retrieval approach for the MRPER, we choose 50 × 10³, 100 × 10³, 150 × 10³, 200 × 10³, 250 × 10³, and 300 × 10³ data files. The MRPER cf:name data retrieval with cf:map = “552901 metallogenic background” and the corresponding search results are presented in Figure 8.

Figure 8

The experimental performance of HBase before and after the optimization: contrast histogram.

The results in Figure 8 demonstrate that there are advantages, compared with the traditional HBase, for the multi-level index evaluation data in geological and MRPER retrieval. This is because the traditional HBase built B+ tree index on the RowKey can only support efficient data queries based on the primary key. Due to the lack of ability of non-primary key index, a full table scan of the component table is required to find the qualified data. In contrast, the multi-level index table can efficiently search through the RowKey table (before a value from the component to the value table as the two level of RowKey), thus avoiding a full table scan of the component table.

4.2 Comparative experiments: HBase and Oracle

4.2.1 Hardware environment

Table 2 shows the hardware environment for testing, in which Oracle uses a single node, while the proposed HBase approach uses a Hadoop cluster deployed on seven computers (three PCs as DataNodes, one PC as a NameNode, and three PCs to configure Zookeeper and HBase).

Table 2

Test database hardware environment

Component	Oracle	Hadoop
Operating system	Windows 10	Red Hat Enterprise Linux Sever
Server count	Home (Chinese-Simplified)	release 6.7 (Santiago)
Server count	One (The Only Node)	Seven (Cluster)
Network bandwidth	100 MB	100 MB
CPU performance	Eight-core	Eight-core
CPU frequency	2.60 GHz	2.4 GHz
RAM	8 G	8 G

4.2.2 Comparison test of data retrieval

In order to compare the data query efficiency of the HBase multi-level index table and a traditional relational database, the geological and MRPER metadata are introduced into the Oracle database and the HBase database, respectively. The amount of data used in the experiment is 0.5 × 10⁶, 1 × 10⁶, 1.5 × 10⁶ , 2 × 10⁶, and 2.5 × 10⁶. In this article, 0.5 × 10⁶ item data are extracted at each time, and experiments based on Oracle and HBase multi-level index table are tested.

From Figure 9, we can see that Oracle has a slight advantage over the proposed HBase approach in retrieving metadata tables of MRPER when there is less data. However, as the amount of data increase, the retrieval time of Oracle increases rapidly. For example, when the amount of data is 2.5 × 10⁶, the time of retrieval increases approximately 37% compared with the time consumption required to process 2 × 10⁶ files. The proposed HBase approach has a slower retrieval speed when the amount of data is small, but with the increase of data volume, its retrieval advantage has been highlighted. The reason for the aforementioned phenomenon is that the system overhead time of the Hadoop distributed cluster exceeds that of Oracle. When the amount of data is small, Oracle naturally takes the lead. When the amount of data is increased, the proposed HBase approach can be directly located to the column, avoiding a full table scan. At the same time, the design of the multi-level index table mainly distributes the burden of retrieval to RowKey, saving the retrieval time. As Oracle is based on row storage in tables, no matter how many fields need to be retrieved, all of the related data in the table need to be scanned. Consequently, as the amount of data increases, the query speed is less than the proposed multi-level HBase approach.

Figure 9

The experimental performance of Oracle and the multi-level HBase approaches: contrast histogram retrieval.

5 Conclusion

Based on the research status of the intelligent retrieval of big data, the HBase data retrieval technology is applied to the field of intelligent retrieval of MRPER. According to the geological data, the related methods are improved, and the feasibility of the method is verified by experiments.

The multi-level index table designed in this article has great advantages over traditional HBase approaches in the data retrieval of geological and MRPER.
When the amount of data is small, the efficiency of traditional relational database retrieval is high. However, the proposed intelligent retrieval method shows great advantages when the amount of data increases dramatically.

Acknowledgments

This article was jointly funded by the NSFC (No. 42171418) and The Key Scientific Research Project of Hunan Provincial Department of Education (No. 22A0332).

Funding information: This article was jointly funded by the NSFC (No. 42171418) and The Key Scientific Research Project of Hunan Provincial Department of Education (No. 22A0332).
Author contributions: Conceptualization: Li Chaokui, Liu Mingxi, and Guo Ruirong; methodology: Li Chaokui; software: Liu Mingxi; validation: Li Chaokui, Zhao Yanan, Yang Wentao, and Zhang Xinchang; writing – original draft preparation: Li Chaokui; writing – review and editing: Liu Mingxi; visualization: Li Chaokui. All authors have read and agreed to the published version of the manuscript.
Conflict of interest: Authors state no conflict of interest.

References

[1] Yan WY. The design and implementation of geological spatial database based on MapGIS and Oracle. XiangTan: Hunan University of Science and Technology; 2016.Search in Google Scholar

[2] Wang WN, Zhao WJ. An efficient image aesthetic analysis system using Hadoop. Signal Process Image Commun. 2015;39(Part C):499–508.10.1016/j.image.2015.07.006Search in Google Scholar

[3] Meng J, Yang MJ. Design and implementation of massive scientific and technological information resources management system based on Hadoop. Sci Technol Manag Res. 2017;37(13):181–6.Search in Google Scholar

[4] Pan J, Biannic YL, Magoulès F. Parallelizing multiple group-by query in share-nothing environment: A MapReduce study case. Acm International Symposium on High Performance Distributed Computing; 2010. p. 856–63.10.1145/1851476.1851599Search in Google Scholar

[5] Hu S, Wu S. Research on preprocessing of vehicle trajectory data based on Hadoop. Lecture Notes Data Eng Commun Technol. 2022;80:1221–8.10.1007/978-3-030-81007-8_140Search in Google Scholar

[6] Khalil MY, Hamad MM. Big data management using Hadoop. J Phys Conf Ser. 2021;1804(1):012109.10.1088/1742-6596/1804/1/012109Search in Google Scholar

[7] Liu JZ. Research of medical imaging storage and retrieval system based on Hadoop and multi-level indexing technology. Chengdu: University of Electronic Science and Technology of China; 2014.Search in Google Scholar

[8] Sun YC. Research on Hadoop-based information retrieval system. Inf Res. 2016;1(8):125–30.Search in Google Scholar

[9] Ma HT. Nested data storage system design and implementation based on HBase. Master dissertation. Zhejiang University; 2015. p. 1–79.Search in Google Scholar

[10] Gao PC, Liu Z, Han F. Accelerating the computation of multi-scale visual curvature for simplifying a large set of polylines with Hadoop. GISci. Remote Sens. 2015;52(3):315–31.10.1080/15481603.2015.1035528Search in Google Scholar

[11] Wan Y, Xiang GL. Research on distributed index clster based on Hadoop and HBase. Inf Technol Inf. 2015(1):102–3.Search in Google Scholar

[12] Chen XP. Research on data generation and index method based on Hbase. Beijing: Beijing University of Posts and Telecommunications; 2013.Search in Google Scholar

[13] Ge W, Luo SM, Zhou WH, Zhao D, Tang Y, Zhou J, et al. HiBase: A hierarchical indexing mechanism and system for efficient HBase query. Chin J Comput. 2016;(1):140–53.Search in Google Scholar

[14] Zou MH. The Design and implementation of full text index for HBase based on lucene. Nanjing: Nanjing University; 2013.Search in Google Scholar

[15] Zuo QC, Ye YQ, Wen H, Song Y, Ge Z, Wang YC, et al. The integrated database model for mineral resources potential evaluation in China. Geol China. 2013;40(6):1968–81.Search in Google Scholar

[16] Wu XN. Study on online analytical processing and data mining in geological environmental data warehouse. Wuhan: China University of Geosciences (Wuhan); 2014.Search in Google Scholar

[17] Chang H, Gao JG, Pan P, Liu XK. Design and development of mineral resource management information system. Mems, Nano and Smart Systems. 2012;403–408:2188–91.10.4028/www.scientific.net/AMR.403-408.2188Search in Google Scholar

[18] Ye J, Zhang L, Guo N, Wang CW. Development and realization of Tibetan mineral resource potential evaluation information management system based on MapGIS K9 Datacenter. Sci Technol Manage Land Resour. 2013;30(6):81–6.Search in Google Scholar

[19] Zuo QC. The technological system for design, development and application of data model and data integration of mineral resources potential evaluation in China. Geol Bull China. 2015;34(12):2334–51.Search in Google Scholar

[20] Zhu JP, Shang PY, Di Q. Data integration of mineral resources Potential evaluation in Jiangsu Province and its application. J Geol. 2015;39(3):400–3.Search in Google Scholar

[21] Nguyen Andrew V, Wynden R, Sun Y. HBase, MapReduce, and integrated data visualization for processing clinical signal Data. AAAI Spring Symposium – Technical Report; 2011. p. 40–4.Search in Google Scholar

[22] Franke G, Morin S, Chebotko A, Abraham J, Brazier P. Distributed semantic web data management in HBase and MySQL Cluster. Proceedings of the 2011 IEEE 4th International Conference on Cloud Computing CLOUD; 2011. p. 105–12.10.1109/CLOUD.2011.19Search in Google Scholar

[23] Jin Y, Deyu T, Yi Z. A distributed storage model for HER based on HBase. Proceedings of the 2011 4th International Conference on Information Management, Innovation Management and Industrial Engineering. ICIII; 2011. p. 369–72.10.1109/ICIII.2011.234Search in Google Scholar

Received: 2022-07-03

Revised: 2023-05-17

Accepted: 2023-06-02

Published Online: 2023-12-31

This work is licensed under the Creative Commons Attribution 4.0 International License.

Articles in the same Issue

https://doi.org/10.1515/geo-2022-0504

Keywords for this article

mineral resources; potential evaluation; results data; Hadoop platform; HBase multi-level index method

Creative Commons

BY 4.0