Defect Data Handling with Python and Perl

1. Theory

• Definition: Defect data handling turns inspection result files, usually KLARF text files, into tables that can be counted, joined to die results, clustered and scored (klarf-reader; pandas).
• KLARF structure: Header records (lot, wafer, step, die pitch, sample centre, inspected area) followed by a DefectRecordSpec that names the columns and a DefectList of semicolon-terminated rows (klarf-reader).
• Coordinates: Each defect is stored by die index and position within the die; wafer coordinates follow from the die pitch and sample centre (klarf-reader; Nanotronics).
• Tables: pandas groups, aggregates and joins defect and die tables (pandas).
• Scripting: Perl remains common for quick text parsing with regular expressions and hashes (perlintro; Doulos).

2. Working Principle

• Parse: Read the DefectRecordSpec to map column names to positions, then read each DefectList row (klarf-reader).
• Count per wafer (pandas):

import pandas as pd
per_wafer = defects.groupby(["lot", "wafer"]).agg(
    n=("DEFECTID", "size"),
    killers=("CLASSNUMBER", lambda c: (c == 4).sum()),  # class 4 as killer (assumed)
)
per_wafer["density"] = per_wafer["n"] / area_inspected

• Join to dies (pandas):

die_def = dies.merge(defects, how="left",
                     on=["wafer", "XINDEX", "YINDEX"], indicator=True)

• Cluster (scikit-learn): DBSCAN(eps=0.5, min_samples=5).fit(xy), with noise labelled -1.
• Perl count of classes: Open the file with a lexical handle, loop over lines, remove newlines with chomp, capture the class field with a regular expression and increment a hash entry keyed by it (perlintro; Doulos).

3. Applications

• Defect density trends: Inputs to the Yield Models and Critical Area page (my reading).
• Kill-ratio studies: Defect-to-die joins (my reading; pandas).
• ADC evaluation: Per-class precision and recall (scikit-learn; Automatic Defect Classification and Machine Learning page).

KLARF Files

Name	Formula	Related Formulae	Variables/Constants
Wafer coordinates from KLARF	$x=X_{index}{p}_x+X_{rel}-x_c,y=Y_{index}{p}_y+Y_{rel}-y_c$	$(p_x,p_y)=\text{DiePitch},(x_c,y_c)=\text{SampleCenterLocation}$	$\begin{array}{r}klarf-readersource;Nanotronics:\\ referencedieat(0,0)withtheoriginat\\ itslower-leftcorner,wafer-centre\\ vector,notchorflatorientation\end{array}$ $\begin{array}{r}(computed:die(3,-2)at10by12\mathrm{mm}\\ pitchwithoffsets(2.5,4)\mathrm{mm}andcentre\\ (1,2)\mathrm{mm}liesat(31.5,-22)\mathrm{mm})\end{array}$
Defect record layout	$\text{DefectRecordSpec}\to \text{column order}$	$\text{DEFECTID, XREL, YREL, XINDEX, YINDEX, DSIZE, CLASSNUMBER,}\dots$	$\begin{array}{r}klarf-reader:columnsfoundbypositionin\\ thespecline,rowsreadafterDefectList,\\ semicolonterminators;klarfrsand\\ jklarfreader:versions1.2and1.8\end{array}$ $\begin{array}{r}(myreading:alwaysreadcolumnsbyname\\ fromthespecline,neverbyfixed\\ position)\end{array}$
Defect density	$D=\frac{N_{defects}}{A_{inspected}}$	$A_{inspected}=\text{AreaPerTest}$	$\begin{array}{r}klarf-reader:AreaPerTestparsedper\\ InspectionTest;YieldModelsandCritical\\ Areapagefor{D}_0\end{array}$ $\begin{array}{r}(computed:140defectson0.0628{\mathrm{m}}^{2}give\\ 2230{\mathrm{m}}^{-2},0.223\mathrm{c}{\mathrm{m}}^{-2})\end{array}$

pandas Tables

Name	Formula	Related Formulae	Variables/Constants
Split-apply-combine	$\text{groupby}\to \text{aggregate, transform, filter}$	$\text{agg}(n=(\text{col},\text{’size’}),\dots )$	$\begin{array}{r}pandas:aggregationreducesgroups,\\ transformationkeepstheindex,filtration\\ keepsordropswholegroups,built-in\\ methodsfasterthanapply\end{array}$ $\begin{array}{r}(myreading:defectsperwaferisan\\ aggregation,awafe{r}^{\prime }sshareofitslota\\ transformation)\end{array}$
Joining defects to die results	$\text{merge}(dies,defects,how=\text{’left’},on=[\text{wafer},x_i,y_i])$	$\text{indicator}\Rightarrow \_merge\in \{\text{left\_only},\text{both}\}$	$\begin{array}{r}pandas.merge:leftjoinkeepseverydie,\\ validatecheckskeyuniqueness,indicator\\ marksunmatchedrows\end{array}$ $\begin{array}{r}(myreading:lef{t}_{o}nlydiesarethe\\ defect-freediesneededforkillratio)\end{array}$
Group-wise checks	$\text{filter}(len>k)$	$\text{transform}(\text{’sum’})\text{broadcasts totals}$	$\begin{array}{r}pandas:filterkeepsgroupspassinga\\ test,transformreturnsalike-indexed\\ result\end{array}$ $\begin{array}{r}(myreading:filterdropswaferswithtoo\\ fewdefectstoclassifyreliably)\end{array}$

Clustering, Scoring and Perl

Name	Formula	Related Formulae	Variables/Constants
Cluster finding	$\text{DBSCAN}(\epsilon ,\text{min\_samples}).\text{fit}(XY)$	$\text{labels\_}=-1\Rightarrow \text{isolated defect}$	$\begin{array}{r}scikit-learn:epsthemostimportant\\ parameter,defaults0.5and5\end{array}$ $\begin{array}{r}(myreading:epsmustbeinthesameunits\\ asthecoordinates,so0.5means0.5\mathrm{mm}if\\ xandyareinmm)\end{array}$
Classifier scoring	$P=\frac{tp}{tp+fp},R=\frac{tp}{tp+fn}$	$\text{average}\in \{\text{micro, macro, weighted}\}$	$\begin{array}{r}scikit-learnprecisio{n}_{r}ecal{l}_{f}scor{e}_{s}upport;Automatic\\ DefectClassificationandMachineLearning\\ page\end{array}$ $\begin{array}{r}(myreading:reportper-classvalues,\\ average=None,forADCbins)\end{array}$
Perl text parsing	$\text{open}\to \text{while}\to \text{chomp}\to \text{regex capture}\to \text{hash count}$	$\text{use strict; use warnings; my}$	$\begin{array}{r}perlintro:strict,warnings,lexicalfile\\ handles,regexcaptures,s///;Doulos:\\ chompandhashcounting\end{array}$ $\begin{array}{r}(myreading:aKLARFclasscountisa\\ regexcaptureintoahash)\end{array}$

Important Questions

Ques: What is in a KLARF file, and how are wafer coordinates recovered?

Header records, a column specification and a defect list. The klarf-reader parser recognises header keywords such as FileVersion, LotID, WaferID, StepID, DiePitch, DieOrigin, SampleCenterLocation, InspectionTest and AreaPerTest. A DefectRecordSpec line names the columns, such as DEFECTID, XREL, YREL, XINDEX, YINDEX, DSIZE and CLASSNUMBER, and the rows follow DefectList, ending in semicolons. It computes $x=X_{index}{p}_x+X_{rel}-x_c$, and likewise for y, from the die pitch and sample centre. Nanotronics places the reference die at (0, 0) with the origin at its lower-left corner. For die (3, -2) at a 10 by 12 $\mathrm{mm}$ pitch, offsets (2.5, 4) $\mathrm{mm}$ and centre (1, 2) $\mathrm{mm}$, the defect lies at (31.5, -22) $\mathrm{mm}$ (computed). Reference: klarf-reader 0.4.3 (Python package by E. Yoong, PyPI and GitHub), source code read: keywords FileVersion, FileTimestamp, InspectionStationID, SampleType, ResultTimestamp, LotID, SampleSize, DeviceID, SetupID, StepID, sample orientation mark type, OrientationMarkLocation, DiePitch, DieOrigin, SampleCenterLocation, WaferID, Slot, InspectionTest, AreaPerTest, DefectRecordSpec, DefectList, SummaryList and SampleTestPlan, defect columns DEFECTID, XREL, YREL, XINDEX, YINDEX, XSIZE, YSIZE, DEFECTAREA, DSIZE, CLASSNUMBER, TEST, CLUSTERNUMBER, ROUGHBINNUMBER, FINEBINNUMBER and IMAGECOUNT located by their position in DefectRecordSpec, lines terminated by semicolons, defect rows read after DefectList, wafer coordinates x = XINDEX * DiePitch.x + XREL - SampleCenterLocation.x and likewise for y, content returned as dataclasses (read from the downloaded package); Nanotronics nSPEC documentation, ‘KLARF export properties’: KLARF reference device at coordinate (0, 0) with the alignment origin at its lower-left corner, WaferCenterX and WaferCenterY as the vector from that origin to the wafer centre, OrientationMarkLocation UP, DOWN, LEFT or RIGHT, sample orientation mark type FLAT or NOTCH, WaferID, SetupID, StepID and SlotNumber fields (fetch-tool summary).

Ques: Which KLARF versions do parsers support, and which statement is right?

Both 1.2 and 1.8 are in use. The klarfrs package contradicts itself: its metadata says it supports only version 1.2, while its description says it supports 1.2 and 1.8 through dynamic column handling. jklarfreader states that it reads the 1.8 format, organised as nested records with lists such as DefectList inside each WaferRecord. klarf-reader locates every defect column by its position in the DefectRecordSpec line rather than by a fixed layout. The body statement of klarfrs is more likely right, because dynamic columns are exactly what a reader needs to handle both versions (my reading). Robust code should read the column order from the specification line in every file (my reading). Reference: klarfrs (PyPI, Rust-based KLARF parser): metadata stating support for version 1.2 only, body stating support for 1.2 and 1.8 via dynamic column handling, fields file_version, lot_id, defect_id, xrel and yrel; jklarfreader (GitHub): reader for the 1.8 format with LotRecord, WaferRecord and DefectList lists (fetch-tool summaries); klarf-reader 0.4.3 (Python package by E. Yoong, PyPI and GitHub), source code read: keywords FileVersion, FileTimestamp, InspectionStationID, SampleType, ResultTimestamp, LotID, SampleSize, DeviceID, SetupID, StepID, sample orientation mark type, OrientationMarkLocation, DiePitch, DieOrigin, SampleCenterLocation, WaferID, Slot, InspectionTest, AreaPerTest, DefectRecordSpec, DefectList, SummaryList and SampleTestPlan, defect columns DEFECTID, XREL, YREL, XINDEX, YINDEX, XSIZE, YSIZE, DEFECTAREA, DSIZE, CLASSNUMBER, TEST, CLUSTERNUMBER, ROUGHBINNUMBER, FINEBINNUMBER and IMAGECOUNT located by their position in DefectRecordSpec, lines terminated by semicolons, defect rows read after DefectList, wafer coordinates x = XINDEX * DiePitch.x + XREL - SampleCenterLocation.x and likewise for y, content returned as dataclasses (read from the downloaded package).

Ques: How are defects per wafer and defect density computed in pandas?

With a group-by aggregation divided by the inspected area. pandas describes group-by as split, apply, combine. Aggregation reduces each group to a value, transformation returns a like-indexed result and filtration keeps or drops whole groups. Named aggregation, such as agg(n=(‘DEFECTID’, ‘size’)), produces readable column names, and size() gives group counts. klarf-reader parses AreaPerTest for each inspection test, which gives the denominator. For example, 140 defects on 0.0628 ${\mathrm{m}}^2$ is 2230 per square metre, or 0.223 $\mathrm{c}{\mathrm{m}}^{-2}$ (computed). That density is the $D_0$ of the Yield Models and Critical Area page, so it must use the inspected area, not the wafer area (my reading). Reference: pandas documentation, ‘Group by: split-apply-combine’: splitting, applying and combining, aggregation, transformation and filtration, df.groupby(‘A’)‘C’,‘D’.sum(), groupby([‘A’,‘B’]).mean(), named aggregation with pd.NamedAgg or (column, aggfunc) tuples, size() for group sizes, transform(‘sum’) broadcasting to rows, filter(lambda x: len(x) > 2) keeping whole groups, built-in methods faster than apply (fetch-tool summary); klarf-reader 0.4.3 (Python package by E. Yoong, PyPI and GitHub), source code read: keywords FileVersion, FileTimestamp, InspectionStationID, SampleType, ResultTimestamp, LotID, SampleSize, DeviceID, SetupID, StepID, sample orientation mark type, OrientationMarkLocation, DiePitch, DieOrigin, SampleCenterLocation, WaferID, Slot, InspectionTest, AreaPerTest, DefectRecordSpec, DefectList, SummaryList and SampleTestPlan, defect columns DEFECTID, XREL, YREL, XINDEX, YINDEX, XSIZE, YSIZE, DEFECTAREA, DSIZE, CLASSNUMBER, TEST, CLUSTERNUMBER, ROUGHBINNUMBER, FINEBINNUMBER and IMAGECOUNT located by their position in DefectRecordSpec, lines terminated by semicolons, defect rows read after DefectList, wafer coordinates x = XINDEX * DiePitch.x + XREL - SampleCenterLocation.x and likewise for y, content returned as dataclasses (read from the downloaded package); Yield Models and Critical Area page of this vault.

Ques: How are defects joined to die test results?

With a left join from dies to defects. pandas.merge with how=‘left’ keeps every key from the left frame, so joining a die table to defects on wafer, XINDEX and YINDEX keeps defect-free dies. With indicator=True, a _merge column marks each row as left_only or both, so dies without defects are flagged directly. The validate option checks key uniqueness, for example ‘many_to_one’. If both key columns hold nulls, pandas matches them to each other, unlike SQL, so missing indices must be cleaned first. In my reading, the left_only dies are the defect-free baseline needed for kill ratio, the subject of the Defect-to-Yield Correlation and Kill Ratio page. Reference: pandas documentation, ‘pandas.merge’: how = inner (default), left, right, outer, cross and anti joins, on, left_on and right_on, indicator adding a _merge column of left_only, right_only or both, validate one_to_one, one_to_many, many_to_one or many_to_many, suffixes (‘_x’, ‘_y’) (fetch-tool summary); klarf-reader 0.4.3 (Python package by E. Yoong, PyPI and GitHub), source code read: keywords FileVersion, FileTimestamp, InspectionStationID, SampleType, ResultTimestamp, LotID, SampleSize, DeviceID, SetupID, StepID, sample orientation mark type, OrientationMarkLocation, DiePitch, DieOrigin, SampleCenterLocation, WaferID, Slot, InspectionTest, AreaPerTest, DefectRecordSpec, DefectList, SummaryList and SampleTestPlan, defect columns DEFECTID, XREL, YREL, XINDEX, YINDEX, XSIZE, YSIZE, DEFECTAREA, DSIZE, CLASSNUMBER, TEST, CLUSTERNUMBER, ROUGHBINNUMBER, FINEBINNUMBER and IMAGECOUNT located by their position in DefectRecordSpec, lines terminated by semicolons, defect rows read after DefectList, wafer coordinates x = XINDEX * DiePitch.x + XREL - SampleCenterLocation.x and likewise for y, content returned as dataclasses (read from the downloaded package).

Ques: How are defect clusters and classifier quality computed in Python?

With scikit-learn’s DBSCAN and precision_recall_fscore_support. DBSCAN takes eps, the neighbourhood distance, and min_samples, the points needed for a core point, defaulting to 0.5 and 5. Defects that belong to no cluster get label -1. In my reading, eps is in the same units as the coordinates, so with millimetre wafer coordinates eps = 0.5 means 0.5 $\mathrm{mm}$. For ADC scoring, precision_recall_fscore_support returns per-class precision $tp/(tp+fp)$, recall $tp/(tp+fn)$, F-score and support. Its average argument chooses micro, macro or weighted means. Per-class values, with average=None, match how bins are reviewed on the Automatic Defect Classification and Machine Learning page, because a macro average can hide one poor killer class behind several good ones (my reading). Reference: scikit-learn 1.0, ‘sklearn.cluster.DBSCAN’: eps (default 0.5) and min_samples (default 5), Euclidean metric, noise labelled -1, example labels 0, 0, 0, 1, 1, -1 (fetch-tool summary); scikit-learn, ‘sklearn.metrics.precision_recall_fscore_support’: precision tp/(tp + fp), recall tp/(tp + fn), F-beta, support, micro, macro and weighted averaging (fetch-tool summary); Automatic Defect Classification and Machine Learning page of this vault.

Ques: What Perl is needed to parse a defect file?

Strict mode, file handles, a read loop, regular expressions and hashes. perlintro starts scripts with use strict and use warnings and declares variables with my. It opens a file through a lexical handle with or die, reporting the system error variable, reads line by line in a while loop over the handle, and closes the handle with or die. Regular expressions capture with parentheses, as in its e-mail example, and substitute with s/foo/bar/g. Doulos shows chomp to remove the trailing newline, the first capture read back as a numbered variable, and counting by incrementing a hash element keyed by that capture. In my reading, counting defects per class in a KLARF file is one regex capture of the CLASSNUMBER field per DefectList row plus a hash increment, then a sorted print of the hash. Reference: perlintro (Perl documentation, Ubuntu manpage): use strict and use warnings, my declarations of scalars, arrays and hashes, open with a lexical handle and die on error (error text in the dollar-bang variable), while loops over the handle, regex captures such as /([^@]+)@(.+)/, substitution s/foo/bar/g, foreach over the keys of a hash, close with die on error (fetch-tool summary); Doulos, ‘Quick start Perl’: scalars marked by a dollar sign, arrays by @, hash elements by key in braces, chomp to remove the trailing newline, line-by-line read loops, regex match with a capture read back as the first capture variable, hash counting by incrementing the element keyed by that capture, keys returning an unordered list (fetch-tool summary).

References

1. E. Yoong, “klarf-reader,” Python package version 0.4.3 (source code read). Available: https://github.com/EricYoong/klarf-reader
2. Nanotronics, “KLARF export properties,” nSPEC documentation v0.25. Available: https://docs.nanotronics.ai/nspec/v0.25.0.0/nspec-ps/klarf-export-properties
3. klarfrs, PyPI project page, and jklarfreader, GitHub repository. Available: https://pypi.org/project/klarfrs/ and https://github.com/btrapp/jklarfreader
4. pandas, “Group by: split-apply-combine,” user guide. Available: https://pandas.pydata.org/docs/user_guide/groupby.html
5. pandas, “pandas.merge,” API reference. Available: https://pandas.pydata.org/docs/reference/api/pandas.merge.html
6. scikit-learn 1.0, “sklearn.cluster.DBSCAN,” documentation. Available: https://scikit-learn.org/1.0/modules/generated/sklearn.cluster.DBSCAN.html
7. scikit-learn, “sklearn.metrics.precision_recall_fscore_support,” documentation. Available: https://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_recall_fscore_support.html
8. Perl 5 documentation, “perlintro: a brief introduction and overview of Perl,” Ubuntu manpage. Available: https://manpages.ubuntu.com/manpages/xenial/man1/perlintro.1.html
9. Doulos, “Quick start Perl.” Available: https://www.doulos.com/knowhow/perl/quick-start-perl/
10. Yield Models and Critical Area page of this vault.
11. Automatic Defect Classification and Machine Learning page of this vault.

Related pages in this vault: Automatic Defect Classification and Machine Learning, Yield Models and Critical Area, Defect Inspection Workflow and Sampling Plans, Wafer-Map Spatial Signature Analysis, Defect-to-Yield Correlation and Kill Ratio.