White Paper: Validation of BTField for One-pass Classification

Edit me

PDF Download

Introduction

This white paper provides an overview of BTField (version 14.3) functionality for Advanced Geophysical Classification (AGC) of dynamic UltraTEM data. We report validation of this functionality via processing of one-pass UltraTEM Portable Classifier and Towed Array Classifier data sets collected at the Ramona Test Site (RATS), San Diego, California (Figure 1 and Figure 2). MRS boundaries for the two data sets were manually defined to include all seeded targets of interest and to ensure 100% data coverage in the data used for classification.

Table 1 summarizes seeded targets of interest (TOI) used for validation of BTField classification in this white paper, and Figure 3 shows the seed ground truth.

Seeded items for which no matching DoD polarizabilities were available (evaluated in UXOLab, e.g. a “MXU-735 Bomb nose plug”) were treated as munitions debris. In addition to the seeded TOI, the site has small metallic clutter including munitions debris (0.50 caliber rounds, 20 mm bolts) and non-munitions related debris (aluminum cans, nails). Based on a review of estimated polarizabilities in UXOLab, we also treated any sources that appeared related to metal in the ground but did not have any associated ground truth information as emplaced clutter.

Target	Min depth (m)	Max depth (m)	# of seeds
155mm	0.66	1.3	2
2.36” HEAT	0.2	0.3	2
2.75” rocket	0.32	0.7	3
37mm	0.08	0.39	5
5” rocket warhead	0.64	0.82	2
60mm mortar	0.4	0.4	1
75mm	0.34	0.49	3
81mm mortar	0.43	0.43	1
90mm	0.63	0.82	3
Large ISO	0.38	1.05	3
M22-R-Grenade-Smoke	0.2	0.2	1
MK2-Grenade	0.13	0.17	2
MK76 (BDU33) 25-lb	0.6	0.6	1
Medium ISO	0.17	0.9	6
Mk23 3-lb	0.33	0.58	4
Mk5 3-lb w/fins	0.45	0.45	1
SCAR Rocket motors	0.29	0.43	2
Small ISO	0.1	0.6	35/41
Total TOI			77/83
Total emplaced clutter			85/87

BTField has previously been validated for Informed Source Selection, here we validate functionality required to classify targets of interest in an ordered dig list generated by matching polarizabilities in a site-specific library. Classification results are compared with previously validated UXOLab software using the RATS data sets. The following sections discuss:

1. the BTField AGC processing workflow; and
2. validation of BTField classification results.

BTField AGC processing workflow

The classification results presented in section 3 were generated with the following processing workflow in BTField. Items shown in green have already been validated as part of ISS processing. Items in blue describe new functionality that has been implemented for classification of one-pass data and is the focus of this white paper.

Data Import

UltraTEM data were imported into BTField from DAGCAP compliant HDF5 V1 files. BTField provides a single solution for both data acquisition and processing, so data can be exported from BTField to HDF5 by the field team and then imported into BTField by data analysts for processing.

Evaluation of Measurement Quality Objectives

Initial and ongoing function tests are evaluated with respect to a previously established reference measurement. IVS measurements are inverted and initial and ongoing location and polarizability accuracy MQOs evaluated against ground truth.

Data cleaning

Data cleaning includes review of production data to remove data that does not meet ongoing production MQOs or which does not support reliable classification.

Data cleaning steps in BTField may include:

• Deletion of data with GPS fix not equal to 4 or with low transmitter current (<15 Amps). Any resulting gaps in positional data are interpolated over distances up to 3m.

• Deletion of spikes, e.g. in IMU data or in EM data (due to external interference).

Background Correction

Data for each Measurement Number (MN - i.e., each unique transmitter-receiver component) are levelled with a demedian filter. Alternatively, a background response defined in a polygon can be used to estimate and remove the mean response for each MN. The latter approach is recommended for areas with a high anomaly density where demedian filtering can overcorrect the data. UltraTEM datasets from RATS presented here were demedian filtered.

Target selection

Target selection is performed by:

• Estimating site specific noise in anomaly-free areas and defining a detection threshold as a multiple of the noise at the detection channel (usually 5x the estimated noise standard deviation).

• Picking targets from a gridded image of the detection channel at the selected detection threshold. Figure 4 and Figure 5 show targets selected in BTField for the two UltraTEM RATS data sets using thresholds defined by 5x the estimated noise. We picked everywhere there is data to allow for picks just outside of the MRS boundary to produce sources within the boundary. For evaluation of classification performance, only sources within the MRS boundaries will be used.

Inversion

Each selected target is inverted with 1, 2 & 3 object inversions in BTField. An inversion mask defines the data around each target that is used in the inversions, here circular masks with a 2m diameter were used.

Figure 7 shows comparisons of polarizabilities estimated in BTField and UXOLab for representative targets in the RATS data sets. There is generally good agreement between polarizabilities estimated with the two software packages, minor observed differences are attributed to details of target selection, noise estimation, masking and optimization routines. In particular, for vertical targets shown in Figure 7, the secondary polarizabilities are poorly constrained by the data and the UXOLab and BTField inversion algorithms can converge to different solutions. As discussed in section Classification, this necessitates a multi-stage classification approach with a stage that uses only the well-constrained primary polarizability to classify targets.

Inversion QC

Models and inversions are failed as per usual QAPP requirements for dynamic data:

• Inversions where the data fit metric is less than < 0.8 are failed. All models from these failed inversions are not used in classification.

• Models that are at horizontal or vertical inversion boundaries are failed.

Figure 8 shows BTField dialogs for failing models with low data fit and models at bounds.

Targets for which all estimated models are failed are treated as “Inconclusive” in the dig list and are flagged for digging. The dig location for an inconclusive target is the pick location.

Classification

An ordered dig list is generated by matching of model polarizabilities against polarizabilities in the site-specific library. Figure 9 shows the library configuration interface in BTField: for classification of the RATS data sets DoD library polarizabilities for each emplaced TOI type were included in the site-specific library.

For each estimated model a polarizability match with each item in the site-specific library is calculated. The match ranges between zero and one, with one indicating a perfect match. The dig list can have multiple stages, with different libraries and match criteria in each stage. For the RATS datasets a two-stage approach was used to classify both data sets:

• In the first stage, matching was done using all three polarizabilities.

• In the second stage, matching was based on only the primary polarizability. This second stage is included to identify targets where secondary polarizabilities are poorly constrained due to low signal to noise ratio or a poorly coupled target orientation (e.g., a vertical target).

Decision points (i.e., the switch between stages 1 and 2, and the stop dig point) are determined by visual inspection of plots of the predicted polarizabilities and best fitting library polarizabilities in dig list order (Figure 10).

For each target, and within each stage, the classifier selects the model that has the highest match metric with all items in the site-specific library. Models that were failed (e.g., due to low data fit) are not used by the classifier. Inconclusive targets on the dig lists are not sorted by polarizability match.

Given selected decision points, the final steps in the classification processing flow in BTField are identification of secondary sources and merging of sources to create a final list of non-redundant (i.e. unique) sources. Secondary sources are sources which meet the decision criteria for a target of interest in one of the classification stages, but for which another source for the same target has already been classified as a target of interest (i.e. category 1 on the dig list). If a secondary source is more than a specified distance (usually 25 cm) from all category 1 sources, then this secondary source is added to category 1. This process accounts for cases where inversion of a single target pick identifies multiple targets of interest.

The final step of merging sources in the dig list is necessary because multiple inversions applied to the same anomaly will resolve the same physical source, and so for intrusive investigations a list of unique sources must be generated. A source is retained on the non-redundant list if there are no other sources earlier on the full (i.e. redundant) dig list that are within the specified horizontal merging distance.

Note that while multi-object inversions can resolve multiple targets of interest in close proximity (i.e. less than 25 cm horizontal separation), the classification process as implemented in BTField does not attempt to resolve these cases. For example, a single source would be selected on the non-redundant dig list for two seeds emplaced within 25 cm. The selected source would correspond to one of the emplaced seeds. Our assumption is that in this type of multi-object case both seeds would still be identified if proper procedures for hole clearance are followed during intrusive investigations. If required on a project, targets of interest in close proximity can be resolved by decreasing the merging distance, but this will increase the number of digs.

Online documentation of the above processing workflow is accessible to BTField users at: https://btdocs.btgeophysics.com/btfield_intro.html.

Validation of BTField classification results

The two RATS data sets were processed independently by different analysts in BTField and UXOLab. To validate BTField for classification processing, we evaluated the BTField dig lists with the following MQOs applicable to DAGCAP DOCs at Aberdeen Proving Ground:

MQO	Acceptance Criteria	Status
All emplaced TOI are classified as TOI in the dig list	All TOI have a dig within 25 cm (horizontal distance) of ground truth location.	Passed. Median/max horizontal offset of digs 6 cm/19 cm (Towed array) and 4 cm/20 cm (Portable) for all emplaced TOI. See Figure 11.
Clutter rejection rate	No more than 40% of emplaced clutter items have a dig within 25 cm (horizontal distance) of ground truth location.	Passed. Dug 11% (Towed Array) and 14% (Portable Classifier) of emplaced clutter.

Figure 12 shows a comparison of BTField and UXOLab classification results for the RATS data sets. To generate these Receiver Operating Characteristic (ROC) curves, we considered any dig that was not within 25 cm of an emplaced target of interest to be a false positive. We also only considered sources that were within the MRS boundaries.

As summarized in the above ROC curves and in Table 2, the results for BTField and UXOLab are similar: all emplaced targets of interest are classified correctly with less than 15% of emplaced clutter dug.

Based on these results, we recommend that BTField be approved by DAGCAP for classification of dynamic UltraTEM data.