Search

News

News & Communication

Data Reply benchmarks technologies for processing geospatial Big Data for Dstl

15 March 2018

Data Reply, the Reply Group company specialising in Big Data and Advanced Analytics, supported the Defence Science and Technology Laboratory, Dstl, in evaluating options for processing large geospatial datasets and providing an evidence-based evaluation of six relevant Big Data technologies. This included benchmarking ingestion, indexing and querying, with latency being the primary objective metric under study.

Dstl, is an executive agency of the UK Ministry of Defence. Through collaborating with academia and innovative businesses, Dstl develops cutting-edge technologies that support UK defence operations. Dstl also provides the UK Government with specialist science and technology research, advice and analysis – much of which is operationally critical and offers potential for technological breakthroughs.

Features that are inherent in the analysis of geospatial data, which Dstl often operates with, cause large scale data processing to be challenging and, often, computationally expensive. In order to gain a deeper understanding of the relative performance of various Big Data technologies, in particular regarding query and ingestion times, Dstl engaged Data Reply to benchmark six prominent Big Data technologies with geospatial data processing capabilities.

Data Reply generated test datasets in the order of 20 billion data points that emulate the structure of geospatial data, including parameterized points and ellipses. Using Google Cloud technology, Data Reply tested six different Big Data technologies (GeoSpark, GeoMesa, Hive, MongoDB, ElasticSearch & Postgres-XL) and benchmarked their data ingestion and query speeds.

The study has provided Dstl with crucial insight into the capabilities and performance of different Big Data technologies. An important goal for Dstl when scoping this engagement was to make the insights from this research available to the public. Data Reply has published the comprehensive report detailing the study and results here:

BENCHMARKING TECHNOLOGIES WITH GEOSPATIAL BIG DATA

Dstl

The Defence Science and Technology Laboratory (Dstl) works to apply cutting-edge science and technology (S&T) to keep UK Armed Forces, and the British people, protected from harm.

Dstl is an Executive Agency of the MOD, run along commercial lines. It is one of the principal government organisations dedicated to S&T in the defence and security field, with four sites; at Porton Down, near Salisbury, Portsdown West, near Portsmouth, Fort Halstead, near Sevenoaks and Alverstoke near Gosport.

www.gov.uk/government/organisations/defence-science-and-technology-laboratory