Wildcard selectors do not correctly search descendants with search_descendants = True

Issues and feature requests
1

OpenRPA version: 1.4.57

Hi, Allan Sorry for the Ai dump below, but after trying around with the AI to figure out what was wrong i wasn’t motivated enough anymore to sumamrize everything myself. The Core essence is that Wildcards in Name get removed to early for search_decendents to work in Windows selectors.

Bug Report:

Summary

In OpenRPA 1.4.57.12, Windows selectors using wildcard characters (*) do not correctly search through descendants of a Win32 TreeView when search_descendants is enabled.

The wildcard functionality itself works correctly, but combining a wildcard Name property with descendant searching produces unexpected results.

An exact Name selector can find a deeply nested TreeItem, while the equivalent wildcard selector cannot. Adding the intermediate TreeItems to the selector makes the wildcard selector work again.

This appears to be a bug or limitation in the Windows selector traversal implementation.

Environment

  • OpenRPA: 1.4.57.12

  • Selector: Windows

  • Application: Ausk32

  • Framework: Win32

  • Control: TreeView / TreeItem

  • search_descendants: True

Expected behavior

When:

"search_descendants": "True"

is enabled, a selector containing:

{
  "Name": "Posteingang*",
  "ControlType": "TreeItem"
}

should search recursively through all descendants of the selected root element and find a TreeItem whose name starts with Posteingang.

For example, given this hierarchy:

KI
└── Postkorb
    └── Posteingang ( 1 )

the following selector should find Posteingang ( 1 ):

[
  {
    "processname": "Ausk32",
    "arguments": "",
    "Selector": "Windows",
    "search_descendants": "True",
    "mouse_over_search": "False"
  },
  {
    "Name": "Posteingang*",
    "ControlType": "TreeItem"
  }
]

Actual behavior

The wildcard selector does not find the descendant when the intermediate hierarchy is omitted.

However, the exact name does work:

[
  {
    "processname": "Ausk32",
    "arguments": "",
    "Selector": "Windows",
    "search_descendants": "True",
    "mouse_over_search": "False"
  },
  {
    "Name": "Posteingang ( 1 )",
    "ControlType": "TreeItem"
  }
]

This finds the element successfully.

This is unexpected because the wildcard selector should be less restrictive, not prevent descendant searching.

Important observation: Wildcards themselves work

The wildcard implementation is not generally broken.

For example:

[
  {
    "processname": "Ausk32",
    "arguments": "",
    "Selector": "Windows",
    "search_descendants": "True",
    "mouse_over_search": "False"
  },
  {
    "Name": "K*",
    "ControlType": "TreeItem"
  }
]

successfully finds:

KI

Therefore * is supported and the wildcard matching itself works.

Further reproduction

The following selector also works:

[
  {
    "processname": "Ausk32",
    "arguments": "",
    "Selector": "Windows",
    "search_descendants": "True",
    "mouse_over_search": "False"
  },
  {
    "Name": "KI",
    "ControlType": "TreeItem"
  },
  {
    "ControlType": "TreeItem"
  },
  {
    "ControlType": "TreeItem"
  }
]

This successfully traverses:

KI
└── Postkorb
    └── Posteingang ( 1 )

even though the second and third TreeItems have no Name specified.

The following also works:

[
  {
    "processname": "Ausk32",
    "arguments": "",
    "Selector": "Windows",
    "search_descendants": "True",
    "mouse_over_search": "False"
  },
  {
    "Name": "KI",
    "ControlType": "TreeItem"
  },
  {
    "Name": "Postkorb*",
    "ControlType": "TreeItem"
  },
  {
    "Name": "Posteingang*",
    "ControlType": "TreeItem"
  }
]

This indicates that explicitly specifying the hierarchy allows the wildcard matching to work.

Source-code investigation

The behavior was investigated against the OpenRPA 1.4.57.12 source code.

1. Wildcard matching is explicitly supported

PatternMatcher.FitsMask() uses StrictMatchPattern():

public static bool FitsMask(string name, string mask)
{
    if (mask == name) return true;
    return PatternMatcher.StrictMatchPattern(mask, name);
}

StrictMatchPattern() supports:

  • * = zero or more characters

  • ? = exactly one character

Therefore wildcard support is intentional.

2. Wildcard properties are removed from the initial UI Automation conditions

In Extensions.GetConditionsWithoutStar():

foreach (var p in item.Properties.Where(
    x => x.Enabled == true &&
         (x.Value != null && !x.Value.Contains("*"))))

Properties containing * are explicitly excluded from the generated UI Automation conditions.

Therefore:

{
  "Name": "Posteingang*",
  "ControlType": "TreeItem"
}

effectively loses the Name condition during the initial candidate search.

Only the non-wildcard properties, such as:

ControlType = TreeItem

remain in the initial condition.

The wildcard Name is subsequently evaluated by WindowsSelectorItem.Match().

3. The selector lookup uses TreeWalker traversal

WindowsSelector.GetElementsWithuiSelectorItem() obtains the conditions using:

var cond = sel.GetConditionsWithoutStar();

and creates a TreeWalker.

When descendant searching is enabled, the implementation traverses using:

_treeWalker.GetFirstChild(...)

and:

_treeWalker.GetNextSibling(...)

The hasStar array is also calculated:

var hasStar = sel.Properties.Where(
    x => x.Enabled == true &&
         (x.Value != null && x.Value.Contains("*"))).ToArray();

but this variable does not appear to alter the subsequent traversal strategy.

Why this causes the problem

There is an important difference between these two selectors.

Exact name

{
  "Name": "Posteingang ( 1 )",
  "ControlType": "TreeItem"
}

The Name property does not contain *, so it remains part of the initial UI Automation condition.

Wildcard name

{
  "Name": "Posteingang*",
  "ControlType": "TreeItem"
}

The Name property contains *, so it is removed from the initial UI Automation condition.

The wildcard comparison is only performed later during selector matching.

This means that the wildcard version cannot use the Name condition to identify the target during the initial traversal.

With the Win32 TreeView hierarchy, this results in the target not being reached unless the intermediate TreeItems are explicitly represented in the selector.

Minimal reproduction

Assume the UI hierarchy is:

KI
└── Postkorb
    └── Posteingang ( 1 )

:cross_mark: Does not find target

[
  {
    "processname": "Ausk32",
    "arguments": "",
    "Selector": "Windows",
    "search_descendants": "True",
    "mouse_over_search": "False"
  },
  {
    "Name": "Posteingang*",
    "ControlType": "TreeItem"
  }
]

:white_check_mark: Finds target

[
  {
    "processname": "Ausk32",
    "arguments": "",
    "Selector": "Windows",
    "search_descendants": "True",
    "mouse_over_search": "False"
  },
  {
    "Name": "Posteingang ( 1 )",
    "ControlType": "TreeItem"
  }
]

:white_check_mark: Finds target

[
  {
    "processname": "Ausk32",
    "arguments": "",
    "Selector": "Windows",
    "search_descendants": "True",
    "mouse_over_search": "False"
  },
  {
    "Name": "KI",
    "ControlType": "TreeItem"
  },
  {
    "ControlType": "TreeItem"
  },
  {
    "ControlType": "TreeItem"
  }
]

:white_check_mark: Finds target

[
  {
    "processname": "Ausk32",
    "arguments": "",
    "Selector": "Windows",
    "search_descendants": "True",
    "mouse_over_search": "False"
  },
  {
    "Name": "KI",
    "ControlType": "TreeItem"
  },
  {
    "Name": "Postkorb*",
    "ControlType": "TreeItem"
  },
  {
    "Name": "Posteingang*",
    "ControlType": "TreeItem"
  }
]

Impact

This makes wildcard selectors unreliable for dynamic/nested Win32 TreeViews.

For example, if the number in:

Posteingang ( 1 )

changes dynamically:

Posteingang ( 1 )
Posteingang ( 2 )
Posteingang ( 3 )
...

the expected solution would be:

Name = Posteingang*

with:

search_descendants = True

However, the user must currently either:

  1. know and specify the complete hierarchy, or

  2. use a less specific selector and rely on TreeItem levels.

This defeats much of the purpose of having descendant searching combined with wildcard matching.

Suggested fix

When search_descendants = True, the selector engine should recursively search all descendants even when one or more selector properties contain wildcards.

One possible implementation would be:

  1. Use the non-wildcard properties as an initial optimization/filter.

  2. Recursively enumerate all descendants.

  3. Apply WindowsSelectorItem.Match() to every candidate.

  4. Allow wildcard properties to determine the final match.

Alternatively, the existing descendant-search implementation in WindowsSelectorItem.matches(), which uses FindAllDescendants(Conditions), could potentially be reused/adjusted so that wildcard properties do not prevent recursive descendant traversal.

The important requirement is:

A wildcard property should affect matching, but should not prevent search_descendants = True from recursively traversing the descendant tree.

Expected result after fix

The following selector should find Posteingang ( 1 ) regardless of its depth:

[
  {
    "processname": "Ausk32",
    "arguments": "",
    "Selector": "Windows",
    "search_descendants": "True",
    "mouse_over_search": "False"
  },
  {
    "Name": "Posteingang*",
    "ControlType": "TreeItem"
  }
]

without requiring:

KI → Postkorb → Posteingang

to be explicitly specified.

Conclusion

This does not appear to be a problem with wildcard syntax.

Wildcard matching works correctly in OpenRPA, as demonstrated by K* successfully matching KI.

The issue appears to be the interaction between:

  • wildcard selector properties,

  • GetConditionsWithoutStar(),

  • search_descendants = True,

  • and the TreeWalker-based Windows selector traversal.

The current behavior means that search_descendants = True does not reliably provide recursive descendant searching when the selector contains wildcard properties.

Severity: Medium / High depending on use case
Component: OpenRPA.Windows selector engine
Affected UI technology: Win32 TreeView / TreeItem
Version tested: 1.4.57.12